Class: NVDFeedScraper::Feed
- Inherits:
-
Object
- Object
- NVDFeedScraper::Feed
- Defined in:
- lib/nvd_feed_api/feed.rb
Overview
Feed object.
Class Attribute Summary collapse
-
.default_storage_location ⇒ String
Get / set default feed storage location, where will be stored JSON feeds and archives by default.
Instance Attribute Summary collapse
-
#data_format ⇒ String
readonly
The format of the feed, should always be
MITRE
. -
#data_number_of_cves ⇒ Integer
readonly
The number of CVEs of in the feed.
-
#data_timestamp ⇒ Date
readonly
The date of the last update of the feed by the NVD.
-
#data_type ⇒ String
readonly
The type of the feed, should always be
CVE
. -
#data_version ⇒ Float
readonly
The version of the JSON schema of the feed.
-
#gz_url ⇒ String
The URL of the gz archive of the feed.
-
#json_file ⇒ String
readonly
The path of the saved JSON file.
-
#meta ⇒ Meta
readonly
The Meta object of the feed.
-
#meta_url ⇒ String
The URL of the metadata file of the feed.
-
#name ⇒ String
The name of the feed.
-
#updated ⇒ String
The last update date of the feed information on the NVD website.
-
#zip_url ⇒ String
The URL of the zip archive of the feed.
Instance Method Summary collapse
-
#available_cves ⇒ Array<String>
Return a list with the name of all available CVEs in the feed.
-
#cve(*arg_cve) ⇒ Object
Search for CVE in the feed.
-
#download_file(file_url, opts = {}) ⇒ String
protected
Download a file.
-
#download_gz(opts = {}) ⇒ String
Download the gz archive of the feed.
-
#download_zip(opts = {}) ⇒ String
Download the zip archive of the feed.
-
#initialize(name, updated, meta_url, gz_url, zip_url) ⇒ Feed
constructor
A new instance of Feed.
-
#json_pull(opts = {}) ⇒ String
Download the JSON feed and fill the attribute.
-
#meta_pull ⇒ Meta
Create or update the Meta object (fill the attribute).
-
#update!(fresh_feed) ⇒ Boolean
Update the feed.
Constructor Details
#initialize(name, updated, meta_url, gz_url, zip_url) ⇒ Feed
A new instance of Feed.
98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 |
# File 'lib/nvd_feed_api/feed.rb', line 98 def initialize(name, updated, , gz_url, zip_url) # From meta file @name = name @updated = updated @meta_url = @gz_url = gz_url @zip_url = zip_url # do not pull meta and json automatically for speed and memory footprint @meta = nil @json_file = nil # feed data @data_type = nil @data_format = nil @data_version = nil @data_number_of_cves = nil @data_timestamp = nil end |
Class Attribute Details
.default_storage_location ⇒ String
Get / set default feed storage location, where will be stored JSON feeds and archives by default.
19 20 21 |
# File 'lib/nvd_feed_api/feed.rb', line 19 def default_storage_location @default_storage_location end |
Instance Attribute Details
#data_format ⇒ String (readonly)
Return nil if not previously loaded by #json_pull.
Returns the format of the feed, should always be MITRE
.
78 79 80 |
# File 'lib/nvd_feed_api/feed.rb', line 78 def data_format @data_format end |
#data_number_of_cves ⇒ Integer (readonly)
Return nil if not previously loaded by #json_pull.
Returns the number of CVEs of in the feed.
86 87 88 |
# File 'lib/nvd_feed_api/feed.rb', line 86 def data_number_of_cves @data_number_of_cves end |
#data_timestamp ⇒ Date (readonly)
Return nil if not previously loaded by #json_pull.
Returns the date of the last update of the feed by the NVD.
90 91 92 |
# File 'lib/nvd_feed_api/feed.rb', line 90 def @data_timestamp end |
#data_type ⇒ String (readonly)
Return nil if not previously loaded by #json_pull.
Returns the type of the feed, should always be CVE
.
74 75 76 |
# File 'lib/nvd_feed_api/feed.rb', line 74 def data_type @data_type end |
#data_version ⇒ Float (readonly)
Return nil if not previously loaded by #json_pull.
Returns the version of the JSON schema of the feed.
82 83 84 |
# File 'lib/nvd_feed_api/feed.rb', line 82 def data_version @data_version end |
#gz_url ⇒ String
Returns the URL of the gz archive of the feed.
41 42 43 |
# File 'lib/nvd_feed_api/feed.rb', line 41 def gz_url @gz_url end |
#json_file ⇒ String (readonly)
Return nil if not previously loaded by #json_pull.
Returns the path of the saved JSON file.
70 71 72 |
# File 'lib/nvd_feed_api/feed.rb', line 70 def json_file @json_file end |
#meta ⇒ Meta (readonly)
Return nil if not previously loaded by #meta_pull. Note that #json_pull also calls #meta_pull.
Returns the Meta object of the feed.
59 60 61 |
# File 'lib/nvd_feed_api/feed.rb', line 59 def @meta end |
#meta_url ⇒ String
Returns the URL of the metadata file of the feed.
36 37 38 |
# File 'lib/nvd_feed_api/feed.rb', line 36 def @meta_url end |
#name ⇒ String
Returns the name of the feed.
26 27 28 |
# File 'lib/nvd_feed_api/feed.rb', line 26 def name @name end |
#updated ⇒ String
Returns the last update date of the feed information on the NVD website.
31 32 33 |
# File 'lib/nvd_feed_api/feed.rb', line 31 def updated @updated end |
#zip_url ⇒ String
Returns the URL of the zip archive of the feed.
46 47 48 |
# File 'lib/nvd_feed_api/feed.rb', line 46 def zip_url @zip_url end |
Instance Method Details
#available_cves ⇒ Array<String>
Return a list with the name of all available CVEs in the feed. Can only be called after #json_pull.
279 280 281 282 283 284 285 286 287 288 289 290 291 292 |
# File 'lib/nvd_feed_api/feed.rb', line 279 def available_cves raise 'json_file is nil, it needs to be populated with json_pull' if @json_file.nil? raise "json_file (#{@json_file}) doesn't exist" unless File.file?(@json_file) doc = Oj::Doc.open(File.read(@json_file)) # Quicker than doc.fetch('/CVE_Items').size cve_names = [] (1..@data_number_of_cves).each do |i| doc.move("/CVE_Items/#{i}") cve_names.push(doc.fetch('cve/CVE_data_meta/ID')) end doc.close return cve_names end |
#cve(cve) ⇒ Hash #cve(cve_arr) ⇒ Array #cve(cve, *) ⇒ Array
implement a CVE Class instead of returning a Hash.
#json_pull is needed before using this method. Remember you're searching only in the current feed.
Search for CVE in the feed.
224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 |
# File 'lib/nvd_feed_api/feed.rb', line 224 def cve(*arg_cve) raise 'json_file is nil, it needs to be populated with json_pull' if @json_file.nil? raise "json_file (#{@json_file}) doesn't exist" unless File.file?(@json_file) return_value = nil raise 'no argument provided, 1 or more expected' if arg_cve.empty? if arg_cve.length == 1 case arg_cve[0] when String raise "bad CVE name (#{arg_cve[0]})" unless /^CVE-[0-9]{4}-[0-9]{4,}$/i.match?(arg_cve[0]) doc = Oj::Doc.open(File.read(@json_file)) # Quicker than doc.fetch('/CVE_Items').size (1..@data_number_of_cves).each do |i| if arg_cve[0].upcase == doc.fetch("/CVE_Items/#{i}/cve/CVE_data_meta/ID") return_value = doc.fetch("/CVE_Items/#{i}") break end end doc.close when Array return_value = [] # Sorting CVE can allow us to parse quicker # Upcase to be sure include? works cves_to_find = arg_cve[0].map(&:upcase).sort raise 'one of the provided arguments is not a String' unless cves_to_find.all? { |x| x.is_a?(String) } raise 'bad CVE name' unless cves_to_find.all? { |x| /^CVE-[0-9]{4}-[0-9]{4,}$/i.match?(x) } doc = Oj::Doc.open(File.read(@json_file)) # Quicker than doc.fetch('/CVE_Items').size (1..@data_number_of_cves).each do |i| doc.move("/CVE_Items/#{i}") cve_id = doc.fetch('cve/CVE_data_meta/ID') if cves_to_find.include?(cve_id) return_value.push(doc.fetch) cves_to_find.delete(cve_id) elsif cves_to_find.empty? break end end raise "#{cves_to_find.join(', ')} are unexisting CVEs in this feed" unless cves_to_find.empty? else raise "the provided argument (#{arg_cve[0]}) is nor a String or an Array" end else # Overloading a list of arguments as one array argument return_value = cve(arg_cve) end return return_value end |
#download_file(file_url, opts = {}) ⇒ String (protected)
Download a file.
357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 |
# File 'lib/nvd_feed_api/feed.rb', line 357 def download_file(file_url, opts = {}) opts[:destination_path] ||= Feed.default_storage_location opts[:sha256] ||= nil destination_path = opts[:destination_path] destination_path += '/' unless destination_path[-1] == '/' skip_download = false uri = URI(file_url) filename = uri.path.split('/').last destination_file = destination_path + filename if !opts[:sha256].nil? && File.file?(destination_file) # Verify hash to see if it is the latest computed_h = Digest::SHA256.file(destination_file) skip_download = true if opts[:sha256].casecmp(computed_h.hexdigest).zero? end unless skip_download res = Net::HTTP.get_response(uri) raise "#{file_url} ended with #{res.code} #{res.}" unless res.is_a?(Net::HTTPSuccess) File.binwrite(destination_file, res.body) end return destination_file end |
#download_gz(opts = {}) ⇒ String
Download the gz archive of the feed.
132 133 134 |
# File 'lib/nvd_feed_api/feed.rb', line 132 def download_gz(opts = {}) download_file(@gz_url, opts) end |
#download_zip(opts = {}) ⇒ String
Download the zip archive of the feed.
142 143 144 |
# File 'lib/nvd_feed_api/feed.rb', line 142 def download_zip(opts = {}) download_file(@zip_url, opts) end |
#json_pull(opts = {}) ⇒ String
Will download and save the zip of the JSON file, unzip and save it. This massively consume time.
Download the JSON feed and fill the attribute.
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 |
# File 'lib/nvd_feed_api/feed.rb', line 151 def json_pull(opts = {}) opts[:destination_path] ||= Feed.default_storage_location skip_download = false destination_path = opts[:destination_path] destination_path += '/' unless destination_path[-1] == '/' filename = URI(@zip_url).path.split('/').last.chomp('.zip') # do not use @json_file for destination_file because of offline loading destination_file = destination_path + filename if File.file?(destination_file) # Verify hash to see if it is the latest computed_h = Digest::SHA256.file(destination_file) skip_download = true if .sha256.casecmp(computed_h.hexdigest).zero? end if skip_download @json_file = destination_file # Set data if @data_type.nil? doc = Oj::Doc.open(File.read(@json_file)) @data_type = doc.fetch('/CVE_data_type') @data_format = doc.fetch('/CVE_data_format') @data_version = doc.fetch('/CVE_data_version').to_f @data_number_of_cves = doc.fetch('/CVE_data_numberOfCVEs').to_i @data_timestamp = Date.strptime(doc.fetch('/CVE_data_timestamp'), '%FT%RZ') doc.close end else zip_path = download_zip(opts) Archive::Zip.open(zip_path) do |z| z.extract(destination_path, flatten: true) end @json_file = zip_path.chomp('.zip') # Verify hash integrity computed_h = Digest::SHA256.file(@json_file) raise "File corruption: #{@json_file}" unless .sha256.casecmp(computed_h.hexdigest).zero? # update data doc = Oj::Doc.open(File.read(@json_file)) @data_type = doc.fetch('/CVE_data_type') @data_format = doc.fetch('/CVE_data_format') @data_version = doc.fetch('/CVE_data_version').to_f @data_number_of_cves = doc.fetch('/CVE_data_numberOfCVEs').to_i @data_timestamp = Date.strptime(doc.fetch('/CVE_data_timestamp'), '%FT%RZ') doc.close end return @json_file end |
#meta_pull ⇒ Meta
Create or update the Meta object (fill the attribute).
119 120 121 122 123 124 |
# File 'lib/nvd_feed_api/feed.rb', line 119 def = NVDFeedScraper::Meta.new(@meta_url) .parse # update @meta @meta = end |
#update!(fresh_feed) ⇒ Boolean
Is not intended to be used directly, use NVDFeedScraper#update_feeds instead.
Update the feed
385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 |
# File 'lib/nvd_feed_api/feed.rb', line 385 def update!(fresh_feed) return_value = false raise "#{fresh_feed} is not a Feed" unless fresh_feed.is_a?(Feed) # update attributes if updated != fresh_feed.updated self.name = fresh_feed.name self.updated = fresh_feed.updated self. = fresh_feed. self.gz_url = fresh_feed.gz_url self.zip_url = fresh_feed.zip_url # update if @meta was set unless @meta.nil? # update if @json_file was set, this will also update @data_* json_pull unless @json_file.nil? return_value = true end return return_value end |