I have the following code, but it takes approximately 3 minutes to process 1000 records. In production I am expecting to have 1 000 000 of records and this performance is unacceptable to process such amount of records. Any idea how to make this faster? I am new with Rails, so still learning the things on the go.
In the example below, I am trying to iterate all products for given supplier and if the product item_id is not in the xml feed, include the product id into array which I will be iterating in the next step and marking the products as "archived / inactive". The problem is mainly with the first part of the code, which takes too much time to process.
self.products.where( :archived => false ).find_each do |p|
archive = !@xml_feed.css("ITEM_ID").to_s.downcase.include?("<item_id>#{p.item_id}</item_id>")
archived_product_ids << p.id if archive
end
if archived_product_ids.size > 0
# update all archived products
Product.where('id IN (?)', archived_product_ids).update_all( :archived => true, :archived_at => Time.now, :active => false )
logger.info "Products #{archived_product_ids.to_s} has been archived and deactivated."
end
This is the output in my console where you can see 3 minutes between processing each 1000 records:
[2015-08-31T22:28:18.090063 #28332] DEBUG -- : Product Load (5.0ms) SELECT "products".* FROM "products" WHERE "products"."supplier_id" = $1 AND "products"."archived" = $2 ORDER BY "products"."id" ASC LIMIT 1000 [["supplier_id", 2], ["archived", "f"]]
[2015-08-31T22:31:14.767496 #28332] DEBUG -- : Product Load (5.3ms) SELECT "products".* FROM "products" WHERE "products"."supplier_id" = $1 AND "products"."archived" = $2 AND ("products"."id" > 2513) ORDER BY "products"."id" ASC LIMIT 1000 [["supplier_id", 2], ["archived", "f"]]
via Chebli Mohamed
Aucun commentaire:
Enregistrer un commentaire