*** Log file does not exist: /usr/local/airflow/logs/lipham_shopify_promotion_and_discount_push/push_discount_to_shopify/2020-11-06T07:58:22.153427+00:00/1.log *** Fetching from: http://c07c705fd854:8793/log/lipham_shopify_promotion_and_discount_push/push_discount_to_shopify/2020-11-06T07:58:22.153427+00:00/1.log [2020-11-06 08:00:04,053] {{taskinstance.py:655}} INFO - Dependencies all met for [2020-11-06 08:00:04,092] {{taskinstance.py:655}} INFO - Dependencies all met for [2020-11-06 08:00:04,092] {{taskinstance.py:866}} INFO - -------------------------------------------------------------------------------- [2020-11-06 08:00:04,092] {{taskinstance.py:867}} INFO - Starting attempt 1 of 1 [2020-11-06 08:00:04,092] {{taskinstance.py:868}} INFO - -------------------------------------------------------------------------------- [2020-11-06 08:00:04,115] {{taskinstance.py:887}} INFO - Executing on 2020-11-06T07:58:22.153427+00:00 [2020-11-06 08:00:04,122] {{standard_task_runner.py:53}} INFO - Started process 123 to run task [2020-11-06 08:00:04,266] {{logging_mixin.py:112}} INFO - Running %s on host %s c07c705fd854 [2020-11-06 08:00:04,293] {{dte_operator.py:151}} INFO - PRE EXECUTE [2020-11-06 08:00:04,293] {{dte_operator.py:480}} INFO - > Getting data in storage of previous task [2020-11-06 08:00:04,304] {{dte_operator.py:485}} DEBUG - File name is: /tmp/lipham_shopify_promotion_and_discount_push_2011060758 [2020-11-06 08:00:04,315] {{dte_operator.py:741}} INFO - # Get credentials of Shopify AdminAPI [2020-11-06 08:00:04,315] {{dte_operator.py:112}} INFO - MATCH (:Company {sysid: $company_id})-[:IN_SYNC_WITH]->(shopify:Shopify) WHERE shopify.status = 'ACTIVE' RETURN shopify [2020-11-06 08:00:04,315] {{dte_operator.py:113}} INFO - {'company_id': 11} [2020-11-06 08:00:04,327] {{logging_mixin.py:112}} INFO - [2020-11-06 08:00:04,327] {{base_hook.py:84}} INFO - Using connection to: id: neo4j_read_default. Host: bolt://integration-branch-backend.systum.com:7688, Port: None, Schema: None, Login: neo4j, Password: XXXXXXXX, extra: None [2020-11-06 08:00:04,338] {{dte_operator.py:127}} DEBUG - [{'shopify': {'APIKey': 'e247ebd5354b6d9ec862474f2c3be321', 'password': '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', 'apiVersion': '2020-07', 'domain': 'lpsystumdev.myshopify.com', 'storefrontAccessToken': 'temp_value_to_prevent_exception_on_integration_airflow', 'sharedSecret': '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', 'status': 'ACTIVE'}}] [2020-11-06 08:00:04,338] {{logging_mixin.py:112}} INFO - set up new configuration for neo4j database. [2020-11-06 08:00:04,338] {{discount_operators.py:268}} INFO - START PUSHING PROMOTION AND DISCOUNT TO SHOPIFY [2020-11-06 08:00:04,340] {{discount_operators.py:269}} DEBUG - {'create': [], 'delete': [{'applyDiscountRule': 'All Sales', 'code': 'Create new and nupdate', 'customFields': '[{"shopify_price_rule_id": 837885657285}]', 'customers': [], 'dateCreated': 1604643945.824694, 'description': 'Testing create new promotion', 'discounts': [{'companyId': 11, 'name': 'WORK IS FUN', 'percentRate': 5.0, 'status': 'ACTIVE', 'sysid': 205216}, {'companyId': 11, 'name': '10%', 'percentRate': 10.0, 'status': 'ACTIVE', 'sysid': 1016}, {'companyId': 11, 'description': '', 'name': 'DLMD11', 'percentRate': 12.0, 'status': 'ACTIVE', 'sysid': 1020}, {'companyId': 11, 'name': 'WORKISFUN', 'percentRate': 5.0, 'status': 'ACTIVE', 'sysid': 205217}], 'endDate': 1605286799.0, 'items': [{'customFields': '[{"shopify_variant_id": ' '37085697310917}, {"qbo_item_id": ' '"1197"}]', 'sysid': 53922}, {'customFields': '[{"shopify_variant_id": ' '37085697278149}, {"qbo_item_id": ' '"1196"}]', 'sysid': 53923}], 'minValue': 0.0, 'startDate': 1604595600.0, 'status': 'DELETED', 'sysid': 1150, 'third_party_id': 837885657285, 'type': 'DISCOUNT_PERCENT'}], 'update': []} [2020-11-06 08:00:04,341] {{discount_operators.py:290}} INFO - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - [2020-11-06 08:00:04,342] {{discount_operators.py:291}} DEBUG - {'applyDiscountRule': 'All Sales', 'code': 'Create new and nupdate', 'customFields': '[{"shopify_price_rule_id": 837885657285}]', 'customers': [], 'dateCreated': 1604643945.824694, 'description': 'Testing create new promotion', 'discounts': [{'companyId': 11, 'name': 'WORK IS FUN', 'percentRate': 5.0, 'status': 'ACTIVE', 'sysid': 205216}, {'companyId': 11, 'name': '10%', 'percentRate': 10.0, 'status': 'ACTIVE', 'sysid': 1016}, {'companyId': 11, 'description': '', 'name': 'DLMD11', 'percentRate': 12.0, 'status': 'ACTIVE', 'sysid': 1020}, {'companyId': 11, 'name': 'WORKISFUN', 'percentRate': 5.0, 'status': 'ACTIVE', 'sysid': 205217}], 'endDate': 1605286799.0, 'items': [{'customFields': '[{"shopify_variant_id": 37085697310917}, ' '{"qbo_item_id": "1197"}]', 'sysid': 53922}, {'customFields': '[{"shopify_variant_id": 37085697278149}, ' '{"qbo_item_id": "1196"}]', 'sysid': 53923}], 'minValue': 0.0, 'startDate': 1604595600.0, 'status': 'DELETED', 'sysid': 1150, 'third_party_id': 837885657285, 'type': 'DISCOUNT_PERCENT'} [2020-11-06 08:00:04,342] {{base_operator.py:75}} DEBUG - Endpoint: https://lpsystumdev.myshopify.com/admin/api/2020-07/price_rules/837885657285.json [2020-11-06 08:00:04,342] {{base_operator.py:76}} DEBUG - Method: DELETE [2020-11-06 08:00:04,342] {{base_operator.py:77}} DEBUG - Payload: None [2020-11-06 08:00:04,542] {{base_operator.py:86}} ERROR - HTTP Error: 404 Client Error: Not Found for url: https://lpsystumdev.myshopify.com/admin/api/2020-07/price_rules/837885657285.json [2020-11-06 08:00:04,542] {{base_operator.py:91}} ERROR - {'errors': 'Not Found'} [2020-11-06 08:00:04,542] {{discount_operators.py:298}} DEBUG - {'discounts': [{'name': 'WORK IS FUN', 'sysid': 205216, 'percentRate': 5.0, 'status': 'ACTIVE', 'companyId': 11}, {'name': '10%', 'sysid': 1016, 'percentRate': 10.0, 'status': 'ACTIVE', 'companyId': 11}, {'name': 'DLMD11', 'description': '', 'sysid': 1020, 'percentRate': 12.0, 'status': 'ACTIVE', 'companyId': 11}, {'name': 'WORKISFUN', 'sysid': 205217, 'percentRate': 5.0, 'status': 'ACTIVE', 'companyId': 11}], 'items': [{'sysid': 53922, 'customFields': '[{"shopify_variant_id": 37085697310917}, {"qbo_item_id": "1197"}]'}, {'sysid': 53923, 'customFields': '[{"shopify_variant_id": 37085697278149}, {"qbo_item_id": "1196"}]'}], 'customers': [], 'endDate': 1605286799.0, 'applyDiscountRule': 'All Sales', 'description': 'Testing create new promotion', 'sysid': 1150, 'customFields': '[{"shopify_price_rule_id": 837885657285}]', 'minValue': 0.0, 'code': 'Create new and nupdate', 'dateCreated': 1604643945.824694, 'status': 'DELETED', 'type': 'DISCOUNT_PERCENT', 'startDate': 1604595600.0, 'third_party_id': 837885657285} [2020-11-06 08:00:04,542] {{dte_operator.py:788}} ERROR - Failed on deleting price rule on Shopify [2020-11-06 08:00:04,543] {{logging_mixin.py:112}} WARNING - Traceback (most recent call last): [2020-11-06 08:00:04,543] {{logging_mixin.py:112}} WARNING - File "/usr/local/airflow/plugins/operators/customized/shopify/discount_operators.py", line 293, in execute self.delete_price_rule(promotion.get('third_party_id')) [2020-11-06 08:00:04,543] {{logging_mixin.py:112}} WARNING - File "/usr/local/airflow/plugins/operators/customized/shopify/discount_operators.py", line 257, in delete_price_rule raise Exception("Failed on deleting price rule on Shopify") [2020-11-06 08:00:04,543] {{logging_mixin.py:112}} WARNING - Exception: Failed on deleting price rule on Shopify [2020-11-06 08:00:04,543] {{dte_operator.py:789}} ERROR - None [2020-11-06 08:00:04,543] {{dte_operator.py:557}} INFO - Push item [{'sysid': 1150}] back to RateLimitingQueue [lipham.systum.com_shopify_promotion_pushing] [2020-11-06 08:00:04,551] {{logging_mixin.py:112}} INFO - [2020-11-06 08:00:04,551] {{base_hook.py:84}} INFO - Using connection to: id: redis_default. Host: redis, Port: 6379, Schema: None, Login: None, Password: None, extra: XXXXXXXX [2020-11-06 08:00:04,554] {{dte_operator.py:571}} INFO - Successfully push item back to RateLimitingQueue [lipham.systum.com_shopify_promotion_pushing] [2020-11-06 08:00:04,554] {{dte_operator.py:378}} INFO - POST EXECUTE [2020-11-06 08:00:04,554] {{dte_operator.py:379}} DEBUG - Failures on this task: [{'data': {'sysid': 1150}, 'reason': 'Failed on deleting price rule on Shopify', 'sysid': 1150, 'task_id': 'push_discount_to_shopify'}] [2020-11-06 08:00:04,563] {{logging_mixin.py:112}} INFO - [2020-11-06 08:00:04,563] {{file_processing_service.py:13}} INFO - [FileHandler] Acquire lock: lipham_shopify_promotion_and_discount_push_2011060758 [2020-11-06 08:00:05,565] {{dte_operator.py:430}} INFO - > Saving data into storage [2020-11-06 08:00:05,565] {{dte_operator.py:480}} INFO - > Getting data in storage of previous task [2020-11-06 08:00:05,573] {{dte_operator.py:485}} DEBUG - File name is: /tmp/lipham_shopify_promotion_and_discount_push_2011060758 [2020-11-06 08:00:05,573] {{logging_mixin.py:112}} INFO - [2020-11-06 08:00:05,573] {{file_processing_service.py:23}} INFO - [FileHandler] Create json data file: lipham_shopify_promotion_and_discount_push_2011060758 [2020-11-06 08:00:05,574] {{dte_operator.py:443}} INFO - Done [2020-11-06 08:00:05,584] {{taskinstance.py:1048}} INFO - Marking task as SUCCESS.dag_id=lipham_shopify_promotion_and_discount_push, task_id=push_discount_to_shopify, execution_date=20201106T075822, start_date=20201106T080004, end_date=20201106T080005 [2020-11-06 08:00:14,000] {{logging_mixin.py:112}} INFO - [2020-11-06 08:00:13,999] {{local_task_job.py:103}} INFO - Task exited with return code 0