Skip to content
This repository was archived by the owner on Jan 23, 2025. It is now read-only.

[HOLD-ON] Listing detail indexes #50

Merged
merged 80 commits into from
Jul 2, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
977fb7f
challenge-listing'
May 18, 2018
31af4dd
challenge-listing'
May 18, 2018
cb1aae6
challenge detail feeder
deedee May 13, 2018
ce17c2c
Merge pull request #46 from tuxing1986/challenge-listing
skyhit May 20, 2018
1e98cb3
mm-challenge-listing
May 20, 2018
0c720b4
mm-challenge-detail
May 21, 2018
332777c
resolve conflict
skyhit May 21, 2018
080af44
Merge branch 'tuxing1986-mm-challenge-listing' into listing-detail-in…
skyhit May 21, 2018
482aa8c
resolve conflict
skyhit May 21, 2018
401a77b
Merge branch 'tuxing1986-mm-challenge-detail' into listing-detail-ind…
skyhit May 21, 2018
0bba1f1
resolve conflict
skyhit May 21, 2018
bee72cf
Merge branch 'deedee-new_challenge_detail' into listing-detail-indexes
skyhit May 21, 2018
cf99206
populate userIds field
skyhit May 24, 2018
b2c0475
revise logic for populating platforms and technologies
skyhit May 25, 2018
ea9e45c
add pointPrizes to ChallengeListingData
deedee May 27, 2018
553f3d3
Merge pull request #54 from deedee/listing-detail-indexes
skyhit May 28, 2018
9a523b1
Update elasticsearch-feeder-service.yaml
skyhit May 29, 2018
e825790
adding test deployment
sushilshinde May 30, 2018
4e9ff3c
Merge branch 'listing-detail-indexes' of https://github.com/topcoder-…
sushilshinde May 30, 2018
dfe7867
adding test deployment
sushilshinde May 30, 2018
e7863c2
adding test deployment
sushilshinde May 30, 2018
eace33a
Change getTimestamp to use internal date
deedee May 31, 2018
207fd08
Merge pull request #56 from deedee/listing-detail-indexes
skyhit Jun 1, 2018
d153633
remove old jobs
skyhit Jun 5, 2018
d7c708b
simplify logging
skyhit Jun 5, 2018
3187d05
revise
skyhit Jun 5, 2018
c46e476
add logging for challenge listing push
skyhit Jun 5, 2018
64718ad
avoid throwing SupplyException for some cases
skyhit Jun 5, 2018
dcf21f8
fix sql file name
skyhit Jun 5, 2018
d5a5427
disable the MM related load jobs
skyhit Jun 5, 2018
eb6753e
improve the query for speed up
skyhit Jun 5, 2018
e44526a
improve query and remove debug logging
skyhit Jun 5, 2018
8fbf654
improve detail requirement population
skyhit Jun 6, 2018
9f3560e
Merge pull request #57 from topcoder-platform/improve_detail_requirem…
skyhit Jun 6, 2018
fefea37
custom setter to trim string
skyhit Jun 6, 2018
804219b
Merge pull request #58 from topcoder-platform/improve_detail_requirem…
skyhit Jun 6, 2018
56a9baa
add submitterIds on challenge listing index
deedee Jun 7, 2018
0464d28
Merge pull request #62 from deedee/listing-detail-indexes
skyhit Jun 7, 2018
8aaa193
Refactor challengedetail model
deedee Jun 7, 2018
dde1494
add detailRequirement for legacy MM detail index
deedee Jun 7, 2018
ef1ce3b
fix sql/mm-feeder-into-challenges/get_registrants.sql
deedee Jun 7, 2018
7ee7b32
Merge pull request #63 from deedee/listing-detail-indexes-0806
skyhit Jun 8, 2018
521860c
job_cleanup_improve2 (#66)
skyhit Jun 11, 2018
9640cf0
enable by default
skyhit Jun 11, 2018
319ebdc
fix sql file name
skyhit Jun 12, 2018
dbf1bf6
refine the job locker name
skyhit Jun 12, 2018
ed88b5a
reduce the batch for legacyMM
skyhit Jun 12, 2018
a7c9ab9
enable detail login
skyhit Jun 12, 2018
951f7ba
ignore 0 update
skyhit Jun 12, 2018
f3029e9
revise the query for get marathon match winners
skyhit Jun 12, 2018
623a438
disable debug logging
skyhit Jun 12, 2018
885f316
revise queries
skyhit Jun 15, 2018
f6ed52a
fix sql query
skyhit Jun 15, 2018
836d0a1
fix sql
skyhit Jun 15, 2018
b57b46f
implement endpoint for populate challenge synchronzied (#70)
skyhit Jun 19, 2018
faf9500
fix prize ordering problem (#69)
skyhit Jun 19, 2018
533696f
cleanup useless model and also fix date string formatting (#68)
skyhit Jun 19, 2018
013aafd
fix numberOfRegistrants field
skyhit Jun 19, 2018
7559fc0
fix sql query
skyhit Jun 19, 2018
496490d
migration to V4
sushilshinde Jun 19, 2018
37b0739
fix numberOfRegistrants field (#71)
skyhit Jun 19, 2018
0422a32
fix index name
skyhit Jun 19, 2018
baa8b1f
Merge branch 'listing-detail-indexes' of github.com:topcoder-platform…
skyhit Jun 19, 2018
cac1b21
fix ordering problem of submissions, registrants and terms (#72)
skyhit Jun 20, 2018
07badc6
fix order for winners
skyhit Jun 20, 2018
e07189c
Merge branch 'listing-detail-indexes' of github.com:topcoder-platform…
skyhit Jun 20, 2018
739cce6
Fix for marathon matches registrants winners (#73)
skyhit Jun 20, 2018
1d2d761
Merge branch 'listing-detail-indexes' of github.com:topcoder-platform…
skyhit Jun 20, 2018
a6d2e9d
CMCTaskId should be string type
skyhit Jun 20, 2018
033e390
fix for populating challenges into ES using end point (#74)
thiyagu06 Jun 20, 2018
c4d85dc
improve logging and query
skyhit Jun 20, 2018
db9e119
Merge branch 'listing-detail-indexes' of github.com:topcoder-platform…
skyhit Jun 20, 2018
494e69a
support custom es-mappings
skyhit Jun 21, 2018
ba08dd2
Fix for issue#389 (#75)
thiyagu06 Jun 21, 2018
c798c3e
upgrading to v4
sushilshinde Jun 26, 2018
f0a7b92
Merge branch 'listing-detail-indexes' of github.com:topcoder-platform…
sushilshinde Jun 26, 2018
67b4a7c
migrating to v4 - prod
sushilshinde Jun 26, 2018
667f66f
Changing workflow
sushilshinde Jun 26, 2018
b53872c
fix sql query
skyhit Jun 27, 2018
a3fdffe
Merge branch 'dev' into listing-detail-indexes
skyhit Jul 2, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions .deploy/ecs_deploy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ ELASTIC_SEARCH_URL=$(eval "echo \$${ENV}_ELASTIC_SEARCH_URL")
TC_JWT_KEY=$(eval "echo \$${ENV}_TC_JWT_KEY")
REDISSON_JOB_SINGLE_SERVER_ADDRESS=$(eval "echo \$${ENV}_REDISSON_JOB_SINGLE_SERVER_ADDRESS")
LOG_LEVEL=$(eval "echo \$${ENV}_LOG_LEVEL")
CHALLENGES_INDEX_NAME=$(eval "echo \$${ENV}_CHALLENGES_INDEX_NAME")
#APP_NAME
OLTP_USER=$(eval "echo \$${ENV}_OLTP_USER")
OLTP_PW=$(eval "echo \$${ENV}_OLTP_PW")
Expand Down Expand Up @@ -84,7 +83,7 @@ push_ecr_image() {
make_task_def(){
echo "Creating ECS task definition..."
task_template=`cat ecs_task_template.json`
task_def=$(printf "$task_template" $AWS_ACCOUNT_ID $AWS_ECS_SERVICE $AWS_REGION "$AUTH_DOMAIN" $AWS_SIGNING_ENABLED $CHALLENGES_INDEX_NAME $ELASTIC_SEARCH_URL $OLTP_PW "$OLTP_URL" $OLTP_USER $TC_JWT_KEY $REDISSON_JOB_SINGLE_SERVER_ADDRESS $LOG_LEVEL $TAG $AWS_ECS_SERVICE $AWS_ACCOUNT_ID $AWS_ECS_SERVICE)
task_def=$(printf "$task_template" $AWS_ACCOUNT_ID $AWS_ECS_SERVICE $AWS_REGION "$AUTH_DOMAIN" $AWS_SIGNING_ENABLED $ELASTIC_SEARCH_URL $OLTP_PW "$OLTP_URL" $OLTP_USER $TC_JWT_KEY $REDISSON_JOB_SINGLE_SERVER_ADDRESS $LOG_LEVEL $TAG $AWS_ECS_SERVICE $AWS_ACCOUNT_ID $AWS_ECS_SERVICE)
echo $task_def > task_def.json
echo "ECS task definition is created : "
echo $task_def
Expand Down
4 changes: 0 additions & 4 deletions .deploy/ecs_task_template.json
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,6 @@
"name": "AWS_SIGNING_ENABLED",
"value": "%s"
},
{
"name": "CHALLENGES_INDEX_NAME",
"value": "%s"
},
{
"name": "ELASTIC_SEARCH_URL",
"value": "%s"
Expand Down
9 changes: 2 additions & 7 deletions circle.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,9 @@ test:

deployment:
development:
branch: ["dev"]
commands:
- .deploy/ecs_deploy.sh DEV
test:
branch: ["test-env"]
commands:
branch: ["dev", "listing-detail-indexes"]
- .deploy/ecs_deploy.sh TEST
production:
branch: master
branch: ["master"]
commands:
- .deploy/ecs_deploy.sh PROD
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
{
"info": {
"_postman_id": "6eee70d5-764c-428f-8c7a-136193804b81",
"name": "Implement Endpoint To Populate Elasticsearch For The Given Challenges",
"schema": "https://schema.getpostman.com/json/collection/v2.0.0/collection.json"
},
"item": [
{
"name": "View challenge details via ElasticSearch",
"request": {
"method": "GET",
"header": [],
"body": {
"mode": "raw",
"raw": ""
},
"url": "http://cockpit.cloud.topcoder.com:9200/challengesdetail/challenges/30005550"
},
"response": [
{
"id": "9156ce02-ba6b-41bb-8819-f7d69d7e8000",
"name": "View challenge details via ElasticSearch",
"originalRequest": {
"method": "GET",
"header": [],
"body": {
"mode": "raw",
"raw": ""
},
"url": "http://cockpit.cloud.topcoder.com:9200/challengesdetail/challenges/30005532"
},
"status": "OK",
"code": 200,
"_postman_previewlanguage": "json",
"header": [
{
"name": "content-length",
"key": "content-length",
"value": "1296",
"description": "The length of the response body in octets (8-bit bytes)"
},
{
"name": "content-type",
"key": "content-type",
"value": "application/json; charset=UTF-8",
"description": "The mime type of this content"
}
],
"cookie": [],
"body": "{\"_index\":\"challengesdetail\",\"_type\":\"challenges\",\"_id\":\"30005532\",\"_version\":6,\"found\":true,\"_source\":{\"track\":\"DEVELOP\",\"subTrack\":\"ARCHITECTURE\",\"detailRequirements\":\"test\",\"finalSubmissionGuidelines\":\"test\",\"terms\":[{\"termsOfUseId\":21193,\"role\":\"Submitter\",\"agreeabilityType\":\"Electronically-agreeable\",\"title\":\"Standard Terms for TopCoder Competitions v2.0\"},{\"termsOfUseId\":20704,\"role\":\"Primary Screener\",\"agreeabilityType\":\"Electronically-agreeable\",\"title\":\"Reviewer terms of use text 20704\"},{\"termsOfUseId\":20704,\"role\":\"Reviewer\",\"agreeabilityType\":\"Electronically-agreeable\",\"title\":\"Reviewer terms of use text 20704\"},{\"termsOfUseId\":20704,\"role\":\"Aggregator\",\"agreeabilityType\":\"Electronically-agreeable\",\"title\":\"Reviewer terms of use text 20704\"},{\"termsOfUseId\":20704,\"role\":\"Final Reviewer\",\"agreeabilityType\":\"Electronically-agreeable\",\"title\":\"Reviewer terms of use text 20704\"},{\"termsOfUseId\":20794,\"role\":\"Manager\",\"agreeabilityType\":\"Non-agreeable\",\"title\":\"Approved OR Managers - TopCoder Technical Team\"},{\"termsOfUseId\":20893,\"role\":\"Copilot\",\"agreeabilityType\":\"Non-electronically-agreeable\",\"title\":\"MSA\"},{\"termsOfUseId\":20704,\"role\":\"Specification Reviewer\",\"agreeabilityType\":\"Electronically-agreeable\",\"title\":\"Reviewer terms of use text 20704\"}],\"id\":30005532}}"
}
]
},
{
"name": "View challenge listing via ElasticSearch",
"request": {
"method": "GET",
"header": [],
"body": {
"mode": "raw",
"raw": ""
},
"url": "http://cockpit.cloud.topcoder.com:9200/challengeslisting/challenges/30005550"
},
"response": []
},
{
"name": "Push challenges",
"request": {
"method": "PUT",
"header": [
{
"key": "Content-Type",
"value": "application/json"
},
{
"key": "Authorization",
"value": "Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJyb2xlcyI6WyJhZG1pbmlzdHJhdG9yIl0sImlzcyI6Imh0dHBzOi8vYXBpLnRvcGNvZGVyLWRldi5jb20iLCJoYW5kbGUiOiJoZWZmYW4iLCJleHAiOjE3NjYyODkyNDYsInVzZXJJZCI6IjEzMjQ1NiIsImlhdCI6MTQ1MDkyOTI0NiwiZW1haWwiOm51bGwsImp0aSI6IjEzNjljNjAwLWUwYTEtNDUyNS1hN2M3LTU2YmU3ZDgxM2Y1MSJ9.hp5peSoj-fh3KFkskvBpfUFIcJNtsv4zIMFV-D8F3JA"
}
],
"body": {
"mode": "raw",
"raw": "{\r\n \"param\": {\r\n \"challengeIds\": [30005550]\r\n }\r\n}"
},
"url": "http://localhost:8083/v4/esfeeder/challenges"
},
"response": []
}
]
}
109 changes: 109 additions & 0 deletions docs/Job Cleanup and Improvement.postman_collection.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
{
"info": {
"_postman_id": "d1083801-f6dd-4220-a6c6-55ddab035754",
"name": "Job Cleanup and Improvement",
"schema": "https://schema.getpostman.com/json/collection/v2.0.0/collection.json"
},
"item": [
{
"name": "View challenges in challenges listing index",
"request": {
"method": "GET",
"header": [],
"body": {
"mode": "raw",
"raw": ""
},
"url": "http://cockpit.cloud.topcoder.com:9200/challengeslisting/challenges/30005520"
},
"response": []
},
{
"name": "View challenges in challenges details index",
"request": {
"method": "GET",
"header": [],
"body": {
"mode": "raw",
"raw": ""
},
"url": "http://cockpit.cloud.topcoder.com:9200/challengesdetail/challenges/30005520"
},
"response": []
},
{
"name": "View marachon match in challenges listing index",
"request": {
"method": "GET",
"header": [],
"body": {
"mode": "raw",
"raw": ""
},
"url": "http://cockpit.cloud.topcoder.com:9200/challengeslisting/challenges/13675"
},
"response": []
},
{
"name": "View marathon match in challenges details index",
"request": {
"method": "GET",
"header": [],
"body": {
"mode": "raw",
"raw": ""
},
"url": "http://cockpit.cloud.topcoder.com:9200/challengeslisting/challenges/13675"
},
"response": []
},
{
"name": "View marathon match",
"request": {
"method": "GET",
"header": [],
"body": {
"mode": "raw",
"raw": ""
},
"url": "http://cockpit.cloud.topcoder.com:9200/mmatches/mmatches/_search"
},
"response": []
},
{
"name": "View single round match",
"request": {
"method": "GET",
"header": [],
"body": {
"mode": "raw",
"raw": ""
},
"url": "http://cockpit.cloud.topcoder.com:9200/srms/srms/_search"
},
"response": []
}
],
"event": [
{
"listen": "prerequest",
"script": {
"id": "d61f2275-4803-4bac-9787-1bfd6c2540e8",
"type": "text/javascript",
"exec": [
""
]
}
},
{
"listen": "test",
"script": {
"id": "40db868c-a50a-4b47-a1cd-73f959da9145",
"type": "text/javascript",
"exec": [
""
]
}
}
]
}
70 changes: 70 additions & 0 deletions docs/challenges-listing-feeder.postman_collection.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
{
"info": {
"_postman_id": "3fce3179-2893-4357-a56f-b07f1ce9785a",
"name": "challenges-listing-feeder",
"schema": "https://schema.getpostman.com/json/collection/v2.0.0/collection.json"
},
"item": [
{
"name": "View all challenges listing via ElasticSearch",
"request": {
"method": "GET",
"header": [],
"body": {
"mode": "raw",
"raw": ""
},
"url": "http://cockpit.cloud.topcoder.com:9200/challengeslisting/challengeslisting/_search"
},
"response": []
},
{
"name": "View challenge via ElasticSearch(assembly with checkpoints)",
"request": {
"method": "GET",
"header": [],
"body": {
"mode": "raw",
"raw": ""
},
"url": "http://cockpit.cloud.topcoder.com:9200/challengeslisting/challengeslisting/30005571"
},
"response": []
},
{
"name": "View challenge via ElasticSearch(wireframe with winner)",
"request": {
"method": "GET",
"header": [],
"body": {
"mode": "raw",
"raw": ""
},
"url": "http://cockpit.cloud.topcoder.com:9200/challengeslisting/challengeslisting/30005528"
},
"response": []
}
],
"event": [
{
"listen": "prerequest",
"script": {
"id": "ee05cdd3-c989-46d1-b3b5-871634dca591",
"type": "text/javascript",
"exec": [
""
]
}
},
{
"listen": "test",
"script": {
"id": "be4c2dd3-d241-4c45-891e-001dd4cc6eb1",
"type": "text/javascript",
"exec": [
""
]
}
}
]
}
70 changes: 70 additions & 0 deletions docs/mm-feeder-into-challenge-detail.postman_collection.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
{
"info": {
"_postman_id": "0e8361f0-20ce-4218-969e-adb4d5481588",
"name": "mm-feeder-into-challenge-detail",
"schema": "https://schema.getpostman.com/json/collection/v2.0.0/collection.json"
},
"item": [
{
"name": "View MM to challenge detail via ElasticSearch with registrant and submitter",
"request": {
"method": "GET",
"header": [],
"body": {
"mode": "raw",
"raw": ""
},
"url": "http://cockpit.cloud.topcoder.com:9200/challengesdetail/challengesdetail/13675"
},
"response": []
},
{
"name": "View MM to challenge listing via ElasticSearch without registrant and submitter",
"request": {
"method": "GET",
"header": [],
"body": {
"mode": "raw",
"raw": ""
},
"url": "http://cockpit.cloud.topcoder.com:9200/challengesdetail/challengesdetail/13673"
},
"response": []
},
{
"name": "View MM to challenge model via ElasticSearch",
"request": {
"method": "GET",
"header": [],
"body": {
"mode": "raw",
"raw": ""
},
"url": "http://cockpit.cloud.topcoder.com:9200/challenges/challenges/13675"
},
"response": []
}
],
"event": [
{
"listen": "prerequest",
"script": {
"id": "d61f2275-4803-4bac-9787-1bfd6c2540e8",
"type": "text/javascript",
"exec": [
""
]
}
},
{
"listen": "test",
"script": {
"id": "40db868c-a50a-4b47-a1cd-73f959da9145",
"type": "text/javascript",
"exec": [
""
]
}
}
]
}
Loading