Skip to content
This repository was archived by the owner on Jan 23, 2025. It is now read-only.

Commit 68bb941

Browse files
skyhitskyhit
authored
[HOLD-ON] Listing detail indexes (#50)
* challenge-listing' * challenge-listing' * challenge detail feeder * mm-challenge-listing * mm-challenge-detail * populate userIds field * revise logic for populating platforms and technologies * add pointPrizes to ChallengeListingData * Update elasticsearch-feeder-service.yaml * adding test deployment * adding test deployment * adding test deployment * Change getTimestamp to use internal date * remove old jobs * simplify logging * revise * add logging for challenge listing push * avoid throwing SupplyException for some cases * fix sql file name * disable the MM related load jobs * improve the query for speed up * improve query and remove debug logging * improve detail requirement population * custom setter to trim string * add submitterIds on challenge listing index * Refactor challengedetail model * add detailRequirement for legacy MM detail index * fix sql/mm-feeder-into-challenges/get_registrants.sql * job_cleanup_improve2 (#66) * enable by default * fix sql file name * refine the job locker name * reduce the batch for legacyMM * enable detail login * ignore 0 update * revise the query for get marathon match winners * disable debug logging * revise queries * fix sql query * fix sql * implement endpoint for populate challenge synchronzied (#70) * fix prize ordering problem (#69) * fix prize ordering problem * fix population of numberOfSubmissions * fix for numberOfCheckpointSubmissions field * cleanup useless model and also fix date string formatting (#68) * cleanup useless model and also fix date string formatting * fix variable doc * fix numberOfRegistrants field * fix sql query * migration to V4 * fix numberOfRegistrants field (#71) * fix numberOfRegistrants field * fix sql query * fix index name * fix ordering problem of submissions, registrants and terms (#72) * fix order for winners * Fix for marathon matches registrants winners (#73) * fix for registrants and winners of marathon matches * CMCTaskId should be string type * fix for populating challenges into ES using end point (#74) * improve logging and query * support custom es-mappings * Fix for issue#389 (#75) * upgrading to v4 * migrating to v4 - prod * Changing workflow * fix sql query
1 parent e36cb7b commit 68bb941

File tree

136 files changed

+5226
-3734
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

136 files changed

+5226
-3734
lines changed

‎.deploy/ecs_deploy.sh

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,6 @@ ELASTIC_SEARCH_URL=$(eval "echo \$${ENV}_ELASTIC_SEARCH_URL")
2323
TC_JWT_KEY=$(eval "echo \$${ENV}_TC_JWT_KEY")
2424
REDISSON_JOB_SINGLE_SERVER_ADDRESS=$(eval "echo \$${ENV}_REDISSON_JOB_SINGLE_SERVER_ADDRESS")
2525
LOG_LEVEL=$(eval "echo \$${ENV}_LOG_LEVEL")
26-
CHALLENGES_INDEX_NAME=$(eval "echo \$${ENV}_CHALLENGES_INDEX_NAME")
2726
#APP_NAME
2827
OLTP_USER=$(eval "echo \$${ENV}_OLTP_USER")
2928
OLTP_PW=$(eval "echo \$${ENV}_OLTP_PW")
@@ -84,7 +83,7 @@ push_ecr_image() {
8483
make_task_def(){
8584
echo "Creating ECS task definition..."
8685
task_template=`cat ecs_task_template.json`
87-
task_def=$(printf "$task_template" $AWS_ACCOUNT_ID $AWS_ECS_SERVICE $AWS_REGION "$AUTH_DOMAIN" $AWS_SIGNING_ENABLED $CHALLENGES_INDEX_NAME $ELASTIC_SEARCH_URL $OLTP_PW "$OLTP_URL" $OLTP_USER $TC_JWT_KEY $REDISSON_JOB_SINGLE_SERVER_ADDRESS $LOG_LEVEL $TAG $AWS_ECS_SERVICE $AWS_ACCOUNT_ID $AWS_ECS_SERVICE)
86+
task_def=$(printf "$task_template" $AWS_ACCOUNT_ID $AWS_ECS_SERVICE $AWS_REGION "$AUTH_DOMAIN" $AWS_SIGNING_ENABLED $ELASTIC_SEARCH_URL $OLTP_PW "$OLTP_URL" $OLTP_USER $TC_JWT_KEY $REDISSON_JOB_SINGLE_SERVER_ADDRESS $LOG_LEVEL $TAG $AWS_ECS_SERVICE $AWS_ACCOUNT_ID $AWS_ECS_SERVICE)
8887
echo $task_def > task_def.json
8988
echo "ECS task definition is created : "
9089
echo $task_def

‎.deploy/ecs_task_template.json

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -32,10 +32,6 @@
3232
"name": "AWS_SIGNING_ENABLED",
3333
"value": "%s"
3434
},
35-
{
36-
"name": "CHALLENGES_INDEX_NAME",
37-
"value": "%s"
38-
},
3935
{
4036
"name": "ELASTIC_SEARCH_URL",
4137
"value": "%s"

‎circle.yml

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -12,14 +12,9 @@ test:
1212

1313
deployment:
1414
development:
15-
branch: ["dev"]
16-
commands:
17-
- .deploy/ecs_deploy.sh DEV
18-
test:
19-
branch: ["test-env"]
20-
commands:
15+
branch: ["dev", "listing-detail-indexes"]
2116
- .deploy/ecs_deploy.sh TEST
2217
production:
23-
branch: master
18+
branch: ["master"]
2419
commands:
2520
- .deploy/ecs_deploy.sh PROD

‎docs/Implement Endpoint To Populate Elasticsearch For The Given Challenges.postman_collection.json

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
{
2+
"info": {
3+
"_postman_id": "6eee70d5-764c-428f-8c7a-136193804b81",
4+
"name": "Implement Endpoint To Populate Elasticsearch For The Given Challenges",
5+
"schema": "https://schema.getpostman.com/json/collection/v2.0.0/collection.json"
6+
},
7+
"item": [
8+
{
9+
"name": "View challenge details via ElasticSearch",
10+
"request": {
11+
"method": "GET",
12+
"header": [],
13+
"body": {
14+
"mode": "raw",
15+
"raw": ""
16+
},
17+
"url": "http://cockpit.cloud.topcoder.com:9200/challengesdetail/challenges/30005550"
18+
},
19+
"response": [
20+
{
21+
"id": "9156ce02-ba6b-41bb-8819-f7d69d7e8000",
22+
"name": "View challenge details via ElasticSearch",
23+
"originalRequest": {
24+
"method": "GET",
25+
"header": [],
26+
"body": {
27+
"mode": "raw",
28+
"raw": ""
29+
},
30+
"url": "http://cockpit.cloud.topcoder.com:9200/challengesdetail/challenges/30005532"
31+
},
32+
"status": "OK",
33+
"code": 200,
34+
"_postman_previewlanguage": "json",
35+
"header": [
36+
{
37+
"name": "content-length",
38+
"key": "content-length",
39+
"value": "1296",
40+
"description": "The length of the response body in octets (8-bit bytes)"
41+
},
42+
{
43+
"name": "content-type",
44+
"key": "content-type",
45+
"value": "application/json; charset=UTF-8",
46+
"description": "The mime type of this content"
47+
}
48+
],
49+
"cookie": [],
50+
"body": "{\"_index\":\"challengesdetail\",\"_type\":\"challenges\",\"_id\":\"30005532\",\"_version\":6,\"found\":true,\"_source\":{\"track\":\"DEVELOP\",\"subTrack\":\"ARCHITECTURE\",\"detailRequirements\":\"test\",\"finalSubmissionGuidelines\":\"test\",\"terms\":[{\"termsOfUseId\":21193,\"role\":\"Submitter\",\"agreeabilityType\":\"Electronically-agreeable\",\"title\":\"Standard Terms for TopCoder Competitions v2.0\"},{\"termsOfUseId\":20704,\"role\":\"Primary Screener\",\"agreeabilityType\":\"Electronically-agreeable\",\"title\":\"Reviewer terms of use text 20704\"},{\"termsOfUseId\":20704,\"role\":\"Reviewer\",\"agreeabilityType\":\"Electronically-agreeable\",\"title\":\"Reviewer terms of use text 20704\"},{\"termsOfUseId\":20704,\"role\":\"Aggregator\",\"agreeabilityType\":\"Electronically-agreeable\",\"title\":\"Reviewer terms of use text 20704\"},{\"termsOfUseId\":20704,\"role\":\"Final Reviewer\",\"agreeabilityType\":\"Electronically-agreeable\",\"title\":\"Reviewer terms of use text 20704\"},{\"termsOfUseId\":20794,\"role\":\"Manager\",\"agreeabilityType\":\"Non-agreeable\",\"title\":\"Approved OR Managers - TopCoder Technical Team\"},{\"termsOfUseId\":20893,\"role\":\"Copilot\",\"agreeabilityType\":\"Non-electronically-agreeable\",\"title\":\"MSA\"},{\"termsOfUseId\":20704,\"role\":\"Specification Reviewer\",\"agreeabilityType\":\"Electronically-agreeable\",\"title\":\"Reviewer terms of use text 20704\"}],\"id\":30005532}}"
51+
}
52+
]
53+
},
54+
{
55+
"name": "View challenge listing via ElasticSearch",
56+
"request": {
57+
"method": "GET",
58+
"header": [],
59+
"body": {
60+
"mode": "raw",
61+
"raw": ""
62+
},
63+
"url": "http://cockpit.cloud.topcoder.com:9200/challengeslisting/challenges/30005550"
64+
},
65+
"response": []
66+
},
67+
{
68+
"name": "Push challenges",
69+
"request": {
70+
"method": "PUT",
71+
"header": [
72+
{
73+
"key": "Content-Type",
74+
"value": "application/json"
75+
},
76+
{
77+
"key": "Authorization",
78+
"value": "Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJyb2xlcyI6WyJhZG1pbmlzdHJhdG9yIl0sImlzcyI6Imh0dHBzOi8vYXBpLnRvcGNvZGVyLWRldi5jb20iLCJoYW5kbGUiOiJoZWZmYW4iLCJleHAiOjE3NjYyODkyNDYsInVzZXJJZCI6IjEzMjQ1NiIsImlhdCI6MTQ1MDkyOTI0NiwiZW1haWwiOm51bGwsImp0aSI6IjEzNjljNjAwLWUwYTEtNDUyNS1hN2M3LTU2YmU3ZDgxM2Y1MSJ9.hp5peSoj-fh3KFkskvBpfUFIcJNtsv4zIMFV-D8F3JA"
79+
}
80+
],
81+
"body": {
82+
"mode": "raw",
83+
"raw": "{\r\n \"param\": {\r\n \"challengeIds\": [30005550]\r\n }\r\n}"
84+
},
85+
"url": "http://localhost:8083/v4/esfeeder/challenges"
86+
},
87+
"response": []
88+
}
89+
]
90+
}

‎docs/Job Cleanup and Improvement.postman_collection.json

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
{
2+
"info": {
3+
"_postman_id": "d1083801-f6dd-4220-a6c6-55ddab035754",
4+
"name": "Job Cleanup and Improvement",
5+
"schema": "https://schema.getpostman.com/json/collection/v2.0.0/collection.json"
6+
},
7+
"item": [
8+
{
9+
"name": "View challenges in challenges listing index",
10+
"request": {
11+
"method": "GET",
12+
"header": [],
13+
"body": {
14+
"mode": "raw",
15+
"raw": ""
16+
},
17+
"url": "http://cockpit.cloud.topcoder.com:9200/challengeslisting/challenges/30005520"
18+
},
19+
"response": []
20+
},
21+
{
22+
"name": "View challenges in challenges details index",
23+
"request": {
24+
"method": "GET",
25+
"header": [],
26+
"body": {
27+
"mode": "raw",
28+
"raw": ""
29+
},
30+
"url": "http://cockpit.cloud.topcoder.com:9200/challengesdetail/challenges/30005520"
31+
},
32+
"response": []
33+
},
34+
{
35+
"name": "View marachon match in challenges listing index",
36+
"request": {
37+
"method": "GET",
38+
"header": [],
39+
"body": {
40+
"mode": "raw",
41+
"raw": ""
42+
},
43+
"url": "http://cockpit.cloud.topcoder.com:9200/challengeslisting/challenges/13675"
44+
},
45+
"response": []
46+
},
47+
{
48+
"name": "View marathon match in challenges details index",
49+
"request": {
50+
"method": "GET",
51+
"header": [],
52+
"body": {
53+
"mode": "raw",
54+
"raw": ""
55+
},
56+
"url": "http://cockpit.cloud.topcoder.com:9200/challengeslisting/challenges/13675"
57+
},
58+
"response": []
59+
},
60+
{
61+
"name": "View marathon match",
62+
"request": {
63+
"method": "GET",
64+
"header": [],
65+
"body": {
66+
"mode": "raw",
67+
"raw": ""
68+
},
69+
"url": "http://cockpit.cloud.topcoder.com:9200/mmatches/mmatches/_search"
70+
},
71+
"response": []
72+
},
73+
{
74+
"name": "View single round match",
75+
"request": {
76+
"method": "GET",
77+
"header": [],
78+
"body": {
79+
"mode": "raw",
80+
"raw": ""
81+
},
82+
"url": "http://cockpit.cloud.topcoder.com:9200/srms/srms/_search"
83+
},
84+
"response": []
85+
}
86+
],
87+
"event": [
88+
{
89+
"listen": "prerequest",
90+
"script": {
91+
"id": "d61f2275-4803-4bac-9787-1bfd6c2540e8",
92+
"type": "text/javascript",
93+
"exec": [
94+
""
95+
]
96+
}
97+
},
98+
{
99+
"listen": "test",
100+
"script": {
101+
"id": "40db868c-a50a-4b47-a1cd-73f959da9145",
102+
"type": "text/javascript",
103+
"exec": [
104+
""
105+
]
106+
}
107+
}
108+
]
109+
}

‎docs/Verification_Implement Endpoint To Populate Elasticsearch For The Given Challenges.docx

239 KB
Binary file not shown.

‎docs/Verification_Topcoder ElasticSearch Feeder Service - Way To Populate Challenge-Detail Index For Legacy Marathon Matches v1.0 v1.0.docx

282 KB
Binary file not shown.

‎docs/Verification_Topcoder ElasticSearch Feeder Service - Way To Populate Challenge-Listing Index For Legacy Marathon Matches v1.0.docx

398 KB
Binary file not shown.

‎docs/Verification_Topcoder ElasticSearch Feeder Service - Way To Populate Challenge-Listing Index v1.0.docx

606 KB
Binary file not shown.

‎docs/Verification_Topcoder Elasticsearch Feeder Service - Jobs Cleanup And Improvement version 1.0.docx

716 KB
Binary file not shown.

‎docs/challenges-listing-feeder.postman_collection.json

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
{
2+
"info": {
3+
"_postman_id": "3fce3179-2893-4357-a56f-b07f1ce9785a",
4+
"name": "challenges-listing-feeder",
5+
"schema": "https://schema.getpostman.com/json/collection/v2.0.0/collection.json"
6+
},
7+
"item": [
8+
{
9+
"name": "View all challenges listing via ElasticSearch",
10+
"request": {
11+
"method": "GET",
12+
"header": [],
13+
"body": {
14+
"mode": "raw",
15+
"raw": ""
16+
},
17+
"url": "http://cockpit.cloud.topcoder.com:9200/challengeslisting/challengeslisting/_search"
18+
},
19+
"response": []
20+
},
21+
{
22+
"name": "View challenge via ElasticSearch(assembly with checkpoints)",
23+
"request": {
24+
"method": "GET",
25+
"header": [],
26+
"body": {
27+
"mode": "raw",
28+
"raw": ""
29+
},
30+
"url": "http://cockpit.cloud.topcoder.com:9200/challengeslisting/challengeslisting/30005571"
31+
},
32+
"response": []
33+
},
34+
{
35+
"name": "View challenge via ElasticSearch(wireframe with winner)",
36+
"request": {
37+
"method": "GET",
38+
"header": [],
39+
"body": {
40+
"mode": "raw",
41+
"raw": ""
42+
},
43+
"url": "http://cockpit.cloud.topcoder.com:9200/challengeslisting/challengeslisting/30005528"
44+
},
45+
"response": []
46+
}
47+
],
48+
"event": [
49+
{
50+
"listen": "prerequest",
51+
"script": {
52+
"id": "ee05cdd3-c989-46d1-b3b5-871634dca591",
53+
"type": "text/javascript",
54+
"exec": [
55+
""
56+
]
57+
}
58+
},
59+
{
60+
"listen": "test",
61+
"script": {
62+
"id": "be4c2dd3-d241-4c45-891e-001dd4cc6eb1",
63+
"type": "text/javascript",
64+
"exec": [
65+
""
66+
]
67+
}
68+
}
69+
]
70+
}

‎docs/mm-feeder-into-challenge-detail.postman_collection.json

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
{
2+
"info": {
3+
"_postman_id": "0e8361f0-20ce-4218-969e-adb4d5481588",
4+
"name": "mm-feeder-into-challenge-detail",
5+
"schema": "https://schema.getpostman.com/json/collection/v2.0.0/collection.json"
6+
},
7+
"item": [
8+
{
9+
"name": "View MM to challenge detail via ElasticSearch with registrant and submitter",
10+
"request": {
11+
"method": "GET",
12+
"header": [],
13+
"body": {
14+
"mode": "raw",
15+
"raw": ""
16+
},
17+
"url": "http://cockpit.cloud.topcoder.com:9200/challengesdetail/challengesdetail/13675"
18+
},
19+
"response": []
20+
},
21+
{
22+
"name": "View MM to challenge listing via ElasticSearch without registrant and submitter",
23+
"request": {
24+
"method": "GET",
25+
"header": [],
26+
"body": {
27+
"mode": "raw",
28+
"raw": ""
29+
},
30+
"url": "http://cockpit.cloud.topcoder.com:9200/challengesdetail/challengesdetail/13673"
31+
},
32+
"response": []
33+
},
34+
{
35+
"name": "View MM to challenge model via ElasticSearch",
36+
"request": {
37+
"method": "GET",
38+
"header": [],
39+
"body": {
40+
"mode": "raw",
41+
"raw": ""
42+
},
43+
"url": "http://cockpit.cloud.topcoder.com:9200/challenges/challenges/13675"
44+
},
45+
"response": []
46+
}
47+
],
48+
"event": [
49+
{
50+
"listen": "prerequest",
51+
"script": {
52+
"id": "d61f2275-4803-4bac-9787-1bfd6c2540e8",
53+
"type": "text/javascript",
54+
"exec": [
55+
""
56+
]
57+
}
58+
},
59+
{
60+
"listen": "test",
61+
"script": {
62+
"id": "40db868c-a50a-4b47-a1cd-73f959da9145",
63+
"type": "text/javascript",
64+
"exec": [
65+
""
66+
]
67+
}
68+
}
69+
]
70+
}

0 commit comments

Comments
 (0)