{"issue":{"state":"open","created_at":"2011-12-06T23:06:19Z","updated_at":"2011-12-06T23:06:19Z","user":{"gravatar_id":"42c8c4ab92de3390bcd2f637ef6ca16c","id":223250,"url":"https://api.github.com/users/kmsquire","login":"kmsquire","avatar_url":"https://secure.gravatar.com/avatar/42c8c4ab92de3390bcd2f637ef6ca16c?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png"},"comments":0,"body":"When two vcf files have very different sizes, one buffer might be starved, leaving only one vcf record in its variant buffer.  If a second variant of the same type at the same location is still not loaded, the variants at that position will be filtered incorrectly.\r\n\r\nSolution was to split out updateVariantStructure() from buildVariantStructure(), and use it to keep the buffer full.  Patch below, also available in commit acdbceb116181d90cd5d6acfad1df6971c0dbfc1.\r\n\r\n```diff\r\ndiff --git a/intersect.cpp b/intersect.cpp\r\nindex 092f92e..908d01a 100644\r\n--- a/intersect.cpp\r\n+++ b/intersect.cpp\r\n@@ -80,17 +80,11 @@ void intersect::intersectVcf(vcfHeader& header1, vcfHeader& header2, vcf& v1, va\r\n           // Clear the compared variants from the structure and add the next one from \r\n           // the file into the structure if it is from the same reference sequence.\r\n           var1.variantMap.erase(var1.vmIter);\r\n-          if (v1.variantRecord.referenceSequence == currentReferenceSequence && v1.success) {\r\n-            var1.addVariantToStructure(v1.position, v1.variantRecord);\r\n-            v1.success = v1.getRecord();\r\n-          }\r\n+          v1.success = var1.updateVariantStructure(v1, currentReferenceSequence);\r\n           if (var1.variantMap.size() != 0) {var1.vmIter = var1.variantMap.begin();}\r\n \r\n           var2.variantMap.erase(var2.vmIter);\r\n-          if (v2.variantRecord.referenceSequence == currentReferenceSequence && v2.success) {\r\n-            var2.addVariantToStructure(v2.position, v2.variantRecord);\r\n-            v2.success = v2.getRecord();\r\n-          }\r\n+          v2.success = var2.updateVariantStructure(v2, currentReferenceSequence);\r\n           if (var2.variantMap.size() != 0) {var2.vmIter = var2.variantMap.begin();}\r\n \r\n         // Variant from the first vcf file is at a larger coordinate than that in the\r\n@@ -99,10 +93,7 @@ void intersect::intersectVcf(vcfHeader& header1, vcfHeader& header2, vcf& v1, va\r\n         } else if (var1.vmIter->first > var2.vmIter->first) {\r\n           if (flags.findCommon) {var2.filterUnique();}\r\n           var2.variantMap.erase(var2.vmIter);\r\n-          if (v2.variantRecord.referenceSequence == currentReferenceSequence && v2.success) {\r\n-            var2.addVariantToStructure(v2.position, v2.variantRecord);\r\n-            v2.success = v2.getRecord();\r\n-          }\r\n+          v2.success = var2.updateVariantStructure(v2, currentReferenceSequence);\r\n \r\n           // Reset the iterator if there are still variants in the structure.\r\n           if (var2.variantMap.size() != 0) {\r\n@@ -130,10 +121,7 @@ void intersect::intersectVcf(vcfHeader& header1, vcfHeader& header2, vcf& v1, va\r\n         } else if (var1.vmIter->first < var2.vmIter->first) {\r\n           if (flags.findCommon && !flags.annotate) {var1.filterUnique();}\r\n           var1.variantMap.erase(var1.vmIter);\r\n-          if (v1.variantRecord.referenceSequence == currentReferenceSequence && v1.success) {\r\n-            var1.addVariantToStructure(v1.position, v1.variantRecord);\r\n-            v1.success = v1.getRecord();\r\n-          }\r\n+          v1.success = var1.updateVariantStructure(v1, currentReferenceSequence);\r\n \r\n           // Reset the iterator if there are still variants in the structure.\r\n           if (var1.variantMap.size() != 0) {\r\ndiff --git a/variant.cpp b/variant.cpp\r\nindex d5b677e..b63e1fd 100644\r\n--- a/variant.cpp\r\n+++ b/variant.cpp\r\n@@ -72,35 +72,44 @@ void variant::determineVariantsToProcess(bool snps, bool mnps, bool indels, bool\r\n \r\n // Build up a structure containing variants.\r\n bool variant::buildVariantStructure(vcf& v) {\r\n-  unsigned int count = 0;\r\n \r\n // When variants in the correct reference sequence are found, build the variant\r\n // structure.\r\n-  count = 0;\r\n   string tempReferenceSequence = v.variantRecord.referenceSequence;\r\n \r\n-  // If this is not the first time that this reference sequence has been seen\r\n-  // when building, the structure then the records for this reference sequence\r\n-  // are not contiguous in the vcf file.  For a number of tools (especially\r\n-  // the intersect tool), this will cause erroneous results to be generated.\r\n-  if (referenceSequenceInfo.count(tempReferenceSequence) != 0) {\r\n-    referenceSequenceInfo[tempReferenceSequence].contiguous = false;\r\n-  }\r\n-  while (v.success && count < recordsInMemory && v.variantRecord.referenceSequence == tempReferenceSequence) {\r\n+  if (v.success && variantMap.size() < recordsInMemory) {\r\n \r\n+    // If this is not the first time that this reference sequence has been seen\r\n+    // when building, the structure then the records for this reference sequence\r\n+    // are not contiguous in the vcf file.  For a number of tools (especially\r\n+    // the intersect tool), this will cause erroneous results to be generated.\r\n+    if (referenceSequenceInfo.count(tempReferenceSequence) != 0) {\r\n+      referenceSequenceInfo[tempReferenceSequence].contiguous = false;\r\n+    }\r\n+    else {\r\n     // Update the information about observed reference sequences.\r\n-    if (referenceSequenceInfo.count(tempReferenceSequence) == 0) {\r\n-      referenceSequenceInfo[tempReferenceSequence].numberRecords    = 1;\r\n+      referenceSequenceInfo[tempReferenceSequence].numberRecords    = 0;\r\n       referenceSequenceInfo[tempReferenceSequence].usedInComparison = false;\r\n       referenceSequenceInfo[tempReferenceSequence].contiguous       = true;\r\n-    } else {\r\n-      referenceSequenceInfo[tempReferenceSequence].numberRecords++;\r\n     }\r\n \r\n+    return updateVariantStructure(v, tempReferenceSequence);\r\n+  }\r\n+\r\n+  return v.success;\r\n+}\r\n+\r\n+// Update the structure containing variants.\r\n+bool variant::updateVariantStructure(vcf& v, string& currentReferenceSequence) {\r\n+\r\n+  while (v.success && variantMap.size() < recordsInMemory && v.variantRecord.referenceSequence == currentReferenceSequence) {\r\n+\r\n+    // Update the information about observed reference sequences.\r\n+    referenceSequenceInfo[currentReferenceSequence].numberRecords++;\r\n+\r\n     // Add the variant into the structure.\r\n     addVariantToStructure(v.position, v.variantRecord);\r\n     v.success = v.getRecord(); \r\n-    count++;\r\n   }\r\n \r\n   return v.success;\r\ndiff --git a/variant.h b/variant.h\r\nindex da900b9..bfc9f37 100644\r\n--- a/variant.h\r\n+++ b/variant.h\r\n@@ -124,6 +124,7 @@ class variant {\r\n     void annotateRecordVcf(variant&, int, unsigned int, string&, string&);\r\n     void buildOutputRecord(output&, vcfHeader&);\r\n     bool buildVariantStructure(vcf&);\r\n+    bool updateVariantStructure(vcf&, string&);\r\n     void clearOriginalVariants(vcfHeader&, intFlags&, output&, bool);\r\n     void clearReferenceSequence(vcfHeader&, vcf&, intFlags, string, output&, bool);\r\n     void clearReferenceSequenceBed(vcfHeader&, vcf&, intFlags, string, output&);\r\n```","assignee":null,"id":2468846,"pull_request":{"patch_url":null,"html_url":null,"diff_url":null},"labels":[],"number":4,"title":"Not all variants filtered with multiple records of same variant type at same position (with patch)","closed_at":null,"html_url":"https://github.com/AlistairNWard/vcfCTools/issues/4","url":"https://api.github.com/repos/AlistairNWard/vcfCTools/issues/4","milestone":null},"action":"opened"}