lundi 31 août 2015

How can I increase performance of the active record iteration (rails)

I have the following code, but it takes approximately 3 minutes to process 1000 records. In production I am expecting to have 1 000 000 of records and this performance is unacceptable to process such amount of records. Any idea how to make this faster? I am new with Rails, so still learning the things on the go.

In the example below, I am trying to iterate all products for given supplier and if the product item_id is not in the xml feed, include the product id into array which I will be iterating in the next step and marking the products as "archived / inactive". The problem is mainly with the first part of the code, which takes too much time to process.

self.products.where( :archived => false ).find_each do |p|
   archive = !@xml_feed.css("ITEM_ID").to_s.downcase.include?("<item_id>#{p.item_id}</item_id>")
   archived_product_ids << p.id if archive
end

if archived_product_ids.size > 0
   # update all archived products
   Product.where('id IN (?)', archived_product_ids).update_all( :archived => true, :archived_at => Time.now, :active => false )
   logger.info "Products #{archived_product_ids.to_s} has been archived and deactivated."
end

This is the output in my console where you can see 3 minutes between processing each 1000 records:

[2015-08-31T22:28:18.090063 #28332] DEBUG -- :   Product Load (5.0ms)  SELECT  "products".* FROM "products" WHERE "products"."supplier_id" = $1 AND "products"."archived" = $2  ORDER BY "products"."id" ASC LIMIT 1000  [["supplier_id", 2], ["archived", "f"]]

[2015-08-31T22:31:14.767496 #28332] DEBUG -- :   Product Load (5.3ms)  SELECT  "products".* FROM "products" WHERE "products"."supplier_id" = $1 AND "products"."archived" = $2 AND ("products"."id" > 2513)  ORDER BY "products"."id" ASC LIMIT 1000  [["supplier_id", 2], ["archived", "f"]]



via Chebli Mohamed

Aucun commentaire:

Enregistrer un commentaire