{"head":"6a45bc368a427d786aa7891d41d03a04a97f67f0","size":18,"push_id":47498189,"ref":"refs/heads/master","commits":[{"sha":"ed23b7d5bde0cd0ea4ffdfa77bb8e1d6e33af354","author":{"name":"Chris Beer","email":"711c73f64afdce07b7e38039a96d2224209e9a6c@cbeer.info"},"url":"https://api.github.com/repos/cbeer/blacklight_oai_harvester_demo/commits/ed23b7d5bde0cd0ea4ffdfa77bb8e1d6e33af354","message":"Add RSpec to the project to provide a structure for our unit tests."},{"sha":"3477b5ed9492abb715bcbcd986a6a08e820b1457","author":{"name":"Chris Beer","email":"711c73f64afdce07b7e38039a96d2224209e9a6c@cbeer.info"},"url":"https://api.github.com/repos/cbeer/blacklight_oai_harvester_demo/commits/3477b5ed9492abb715bcbcd986a6a08e820b1457","message":"Generate provider and record scaffolding\n\nGenerated using these commands:\n\n$ rails generate scaffold provider name:string description:text\nendpoint_url:string  metadata_prefix:string set:string\n$ rails generate scaffold record content:text provider:references\n$ rake db:migrate"},{"sha":"6efe19e976586d85e0171aab136664c55e29356e","author":{"name":"Chris Beer","email":"711c73f64afdce07b7e38039a96d2224209e9a6c@cbeer.info"},"url":"https://api.github.com/repos/cbeer/blacklight_oai_harvester_demo/commits/6efe19e976586d85e0171aab136664c55e29356e","message":"Add rspec tests (and factory_girl fixture)\n\nStub out a Provider spec"},{"sha":"a586592727d7274feeb074ede7075195df8632d4","author":{"name":"Chris Beer","email":"711c73f64afdce07b7e38039a96d2224209e9a6c@cbeer.info"},"url":"https://api.github.com/repos/cbeer/blacklight_oai_harvester_demo/commits/a586592727d7274feeb074ede7075195df8632d4","message":"Add sensible defaults for Provider#metadata_prefix\n\nBy default, use the required oai_dc metadata prefix."},{"sha":"092bf7cc9bd2ba7da044cdb15113d103962eaae6","author":{"name":"Chris Beer","email":"711c73f64afdce07b7e38039a96d2224209e9a6c@cbeer.info"},"url":"https://api.github.com/repos/cbeer/blacklight_oai_harvester_demo/commits/092bf7cc9bd2ba7da044cdb15113d103962eaae6","message":"Add the ability to harvest and ingest oai-pmh docs\n\nUse the \"oai\" gem to access OAI-PMH providers, and add\nthe accessor Provider#client.\n\nAdd Provider#each_record to perform an action on each\nrecord in an OAI feed\n\nAdd Provider#consume! that ingests a new record (using #each_record)"},{"sha":"ebcf603371a605e607c4ee57e03345fb2bb27b3e","author":{"name":"Chris Beer","email":"711c73f64afdce07b7e38039a96d2224209e9a6c@cbeer.info"},"url":"https://api.github.com/repos/cbeer/blacklight_oai_harvester_demo/commits/ebcf603371a605e607c4ee57e03345fb2bb27b3e","message":"Customized scaffold views and controller\n\nNow that we are harvesting data, modify the views to link between a Provider and its Records"},{"sha":"d654aca9b85463b6ee5688275d4a8cae291afe11","author":{"name":"Chris Beer","email":"711c73f64afdce07b7e38039a96d2224209e9a6c@cbeer.info"},"url":"https://api.github.com/repos/cbeer/blacklight_oai_harvester_demo/commits/d654aca9b85463b6ee5688275d4a8cae291afe11","message":"Set up an om xml document for oai-pmh oai_dc docs\n\nom (opinionated metadata) is a gem for translating between\nXML schemas and Ruby objects (it is used extensively by Hydra, which is\nalso a Blacklight application)\n\nInstead of accessing the om terminology for the Record#content\nindirectly, we use the Rails 3.x .serialize. By adding a special\nSerializer class, we can access the terminology (and modify, save, etc)\ndirectly on Record#content."},{"sha":"a24cb1a80a25c3eba23f778d658749709d71dc01","author":{"name":"Chris Beer","email":"9806d69c57c8f83fa19a9d5913c84839755383fc@wgbh.org"},"url":"https://api.github.com/repos/cbeer/blacklight_oai_harvester_demo/commits/a24cb1a80a25c3eba23f778d658749709d71dc01","message":"Split OAI-PMH record metadata from the payload\n\nWe want to think about supporting generic OAI-PMH provided content. The\nstructure of an OAI-PMH document is defined in the specification, but\nthe payload of <metadata> is provider specific.\n\nHere, we create the model OaiPmhRecord, which can be mixed into any\nclass that provides the appropriate terminology helpers:\n\n.set_oai_metadata_terminology : add metadata-specific terminology\n\n.add_oai_metadata_terminology_proxies : add top-level terminology\naccessors to the OAI document"},{"sha":"e3115b56de661185971a1db26e49886f5c735ade","author":{"name":"Chris Beer","email":"9806d69c57c8f83fa19a9d5913c84839755383fc@wgbh.org"},"url":"https://api.github.com/repos/cbeer/blacklight_oai_harvester_demo/commits/e3115b56de661185971a1db26e49886f5c735ade","message":"Record#content= must convert content to the OaiDcDocument instance"},{"sha":"23aba50e5ff7594ba2343192f8275a8c3cc52d4f","author":{"name":"Chris Beer","email":"711c73f64afdce07b7e38039a96d2224209e9a6c@cbeer.info"},"url":"https://api.github.com/repos/cbeer/blacklight_oai_harvester_demo/commits/23aba50e5ff7594ba2343192f8275a8c3cc52d4f","message":"Use solrizer to translate OM terms to solr docs\n\nSearching within a database is a drag. Use the solrizer gem (and\nSolrizer::XML::TerminologyBasedSolrizer) to translate our OAI-PMH\nterminologies into documents that can be indexed into solr.\n\nWe'll get around to adding solr shortly."},{"sha":"4c066214471d9293da5535fc7899c192590003aa","author":{"name":"Chris Beer","email":"711c73f64afdce07b7e38039a96d2224209e9a6c@cbeer.info"},"url":"https://api.github.com/repos/cbeer/blacklight_oai_harvester_demo/commits/4c066214471d9293da5535fc7899c192590003aa","message":"Install the blacklight rails engine.\n\nBlacklight provides an interface to a Solr instance. Generated using:\n$ rails g blacklight:install --devise\n\nWe'll customize Blacklight for our index in the next step.\n\nAlso, remove the scaffold css; Blacklight's theme will do."},{"sha":"3b2120f51b91bbfd3efe887252b5815a1ab71da7","author":{"name":"Chris Beer","email":"711c73f64afdce07b7e38039a96d2224209e9a6c@cbeer.info"},"url":"https://api.github.com/repos/cbeer/blacklight_oai_harvester_demo/commits/3b2120f51b91bbfd3efe887252b5815a1ab71da7","message":"Configure blacklight for our custom application\n\nReplace Blacklight demo configuration with OAI-PMH oai_dc terms.\n\nAlso, expand the OaiDcDocument terminology to better index terms that\nBlacklight will use to provide functionality."},{"sha":"f90294f493e7c095d7bf6b5e0604f0473ab697cf","author":{"name":"Chris Beer","email":"9806d69c57c8f83fa19a9d5913c84839755383fc@wgbh.org"},"url":"https://api.github.com/repos/cbeer/blacklight_oai_harvester_demo/commits/f90294f493e7c095d7bf6b5e0604f0473ab697cf","message":"override blacklight's application_name helper"},{"sha":"6619ea4bd8bec751bc48f69a96a303f064e5b41f","author":{"name":"Chris Beer","email":"711c73f64afdce07b7e38039a96d2224209e9a6c@cbeer.info"},"url":"https://api.github.com/repos/cbeer/blacklight_oai_harvester_demo/commits/6619ea4bd8bec751bc48f69a96a303f064e5b41f","message":"Update Record to index records into Solr.\n\nUse a parameterized version of the record identifier\n(//record/identifier) as the solr document unique key.\n\nAnd, add an after_save filter for Record that will index the record into solr\n\nAnd, autocommit any adds to Solr within 50ms"},{"sha":"9f739e9202aacdf2b68ce4f9a12b16abd6709d5d","author":{"name":"Chris Beer","email":"9806d69c57c8f83fa19a9d5913c84839755383fc@wgbh.org"},"url":"https://api.github.com/repos/cbeer/blacklight_oai_harvester_demo/commits/9f739e9202aacdf2b68ce4f9a12b16abd6709d5d","message":"Add Solr!\n\nUse hydra-jetty (a Jetty instance that includes Solr), included as a submodule\n\nAnd, add the jettywrapper gem to manage it, giving us access to rake\ntasks for starting/stopping/etc jetty"},{"sha":"505c6a14d00d2463cb3e514dcedba3cc5e306254","author":{"name":"Chris Beer","email":"9806d69c57c8f83fa19a9d5913c84839755383fc@wgbh.org"},"url":"https://api.github.com/repos/cbeer/blacklight_oai_harvester_demo/commits/505c6a14d00d2463cb3e514dcedba3cc5e306254","message":"Store harvest metadata (to support scheduled crawls)\n\nAdd harvest time and interval data and add task to harvest from providers\n\nAdd 'from' to the oai default parameters to limit crawling to records\nmodified after 'from'"},{"sha":"f33c64e90109e0ca2b3384a1bff4d5d81a16a966","author":{"name":"Chris Beer","email":"9806d69c57c8f83fa19a9d5913c84839755383fc@wgbh.org"},"url":"https://api.github.com/repos/cbeer/blacklight_oai_harvester_demo/commits/f33c64e90109e0ca2b3384a1bff4d5d81a16a966","message":"Remove Record scaffolding. Solr documents are sufficient.\n\nInstead of creating and persisting OAI-PMH harvested records in the\ndatabase, we can just harvest and create documents within Solr."},{"sha":"6a45bc368a427d786aa7891d41d03a04a97f67f0","author":{"name":"Chris Beer","email":"9806d69c57c8f83fa19a9d5913c84839755383fc@wgbh.org"},"url":"https://api.github.com/repos/cbeer/blacklight_oai_harvester_demo/commits/6a45bc368a427d786aa7891d41d03a04a97f67f0","message":"Add a link back to the record provider.\n\nOverride helpers render_index_field_value and\nrender_document_show_field_value to replace the provider_id_i field's\nvalue with a link to the provider."}],"legacy":{"head":"6a45bc368a427d786aa7891d41d03a04a97f67f0","size":18,"push_id":47498189,"shas":[["ed23b7d5bde0cd0ea4ffdfa77bb8e1d6e33af354","chris@cbeer.info","Add RSpec to the project to provide a structure for our unit tests.","Chris Beer"],["3477b5ed9492abb715bcbcd986a6a08e820b1457","chris@cbeer.info","Generate provider and record scaffolding\n\nGenerated using these commands:\n\n$ rails generate scaffold provider name:string description:text\nendpoint_url:string  metadata_prefix:string set:string\n$ rails generate scaffold record content:text provider:references\n$ rake db:migrate","Chris Beer"],["6efe19e976586d85e0171aab136664c55e29356e","chris@cbeer.info","Add rspec tests (and factory_girl fixture)\n\nStub out a Provider spec","Chris Beer"],["a586592727d7274feeb074ede7075195df8632d4","chris@cbeer.info","Add sensible defaults for Provider#metadata_prefix\n\nBy default, use the required oai_dc metadata prefix.","Chris Beer"],["092bf7cc9bd2ba7da044cdb15113d103962eaae6","chris@cbeer.info","Add the ability to harvest and ingest oai-pmh docs\n\nUse the \"oai\" gem to access OAI-PMH providers, and add\nthe accessor Provider#client.\n\nAdd Provider#each_record to perform an action on each\nrecord in an OAI feed\n\nAdd Provider#consume! that ingests a new record (using #each_record)","Chris Beer"],["ebcf603371a605e607c4ee57e03345fb2bb27b3e","chris@cbeer.info","Customized scaffold views and controller\n\nNow that we are harvesting data, modify the views to link between a Provider and its Records","Chris Beer"],["d654aca9b85463b6ee5688275d4a8cae291afe11","chris@cbeer.info","Set up an om xml document for oai-pmh oai_dc docs\n\nom (opinionated metadata) is a gem for translating between\nXML schemas and Ruby objects (it is used extensively by Hydra, which is\nalso a Blacklight application)\n\nInstead of accessing the om terminology for the Record#content\nindirectly, we use the Rails 3.x .serialize. By adding a special\nSerializer class, we can access the terminology (and modify, save, etc)\ndirectly on Record#content.","Chris Beer"],["a24cb1a80a25c3eba23f778d658749709d71dc01","chris_beer@wgbh.org","Split OAI-PMH record metadata from the payload\n\nWe want to think about supporting generic OAI-PMH provided content. The\nstructure of an OAI-PMH document is defined in the specification, but\nthe payload of <metadata> is provider specific.\n\nHere, we create the model OaiPmhRecord, which can be mixed into any\nclass that provides the appropriate terminology helpers:\n\n.set_oai_metadata_terminology : add metadata-specific terminology\n\n.add_oai_metadata_terminology_proxies : add top-level terminology\naccessors to the OAI document","Chris Beer"],["e3115b56de661185971a1db26e49886f5c735ade","chris_beer@wgbh.org","Record#content= must convert content to the OaiDcDocument instance","Chris Beer"],["23aba50e5ff7594ba2343192f8275a8c3cc52d4f","chris@cbeer.info","Use solrizer to translate OM terms to solr docs\n\nSearching within a database is a drag. Use the solrizer gem (and\nSolrizer::XML::TerminologyBasedSolrizer) to translate our OAI-PMH\nterminologies into documents that can be indexed into solr.\n\nWe'll get around to adding solr shortly.","Chris Beer"],["4c066214471d9293da5535fc7899c192590003aa","chris@cbeer.info","Install the blacklight rails engine.\n\nBlacklight provides an interface to a Solr instance. Generated using:\n$ rails g blacklight:install --devise\n\nWe'll customize Blacklight for our index in the next step.\n\nAlso, remove the scaffold css; Blacklight's theme will do.","Chris Beer"],["3b2120f51b91bbfd3efe887252b5815a1ab71da7","chris@cbeer.info","Configure blacklight for our custom application\n\nReplace Blacklight demo configuration with OAI-PMH oai_dc terms.\n\nAlso, expand the OaiDcDocument terminology to better index terms that\nBlacklight will use to provide functionality.","Chris Beer"],["f90294f493e7c095d7bf6b5e0604f0473ab697cf","chris_beer@wgbh.org","override blacklight's application_name helper","Chris Beer"],["6619ea4bd8bec751bc48f69a96a303f064e5b41f","chris@cbeer.info","Update Record to index records into Solr.\n\nUse a parameterized version of the record identifier\n(//record/identifier) as the solr document unique key.\n\nAnd, add an after_save filter for Record that will index the record into solr\n\nAnd, autocommit any adds to Solr within 50ms","Chris Beer"],["9f739e9202aacdf2b68ce4f9a12b16abd6709d5d","chris_beer@wgbh.org","Add Solr!\n\nUse hydra-jetty (a Jetty instance that includes Solr), included as a submodule\n\nAnd, add the jettywrapper gem to manage it, giving us access to rake\ntasks for starting/stopping/etc jetty","Chris Beer"],["505c6a14d00d2463cb3e514dcedba3cc5e306254","chris_beer@wgbh.org","Store harvest metadata (to support scheduled crawls)\n\nAdd harvest time and interval data and add task to harvest from providers\n\nAdd 'from' to the oai default parameters to limit crawling to records\nmodified after 'from'","Chris Beer"],["f33c64e90109e0ca2b3384a1bff4d5d81a16a966","chris_beer@wgbh.org","Remove Record scaffolding. Solr documents are sufficient.\n\nInstead of creating and persisting OAI-PMH harvested records in the\ndatabase, we can just harvest and create documents within Solr.","Chris Beer"],["6a45bc368a427d786aa7891d41d03a04a97f67f0","chris_beer@wgbh.org","Add a link back to the record provider.\n\nOverride helpers render_index_field_value and\nrender_document_show_field_value to replace the provider_id_i field's\nvalue with a link to the provider.","Chris Beer"]],"ref":"refs/heads/master"}}