gallery-dl
gallery-dl copied to clipboard
mikf
[weverse] add extractors
Closes https://github.com/mikf/gallery-dl/issues/3995.
Weverse requires an account for paginated URLs. Its API is not publicly documented.
The API code is inspired by the yt-dlp implementation.
~Speaking of yt-dlp, we could defer to it for media posts, as it supports VOD and live broadcasts as far as I can tell, but I haven't done this yet.~
Some miscellaneous questions:
- I noticed that you recently set
Refererfor requests by default, but it is absolutely mandatory for Weverse. Is it worth setting in the API headers? - The API responses use camelCase. Currently I am adding custom values to the metadata using snake_case. Should I stick to one style instead of mixing them?
Example API responses:
Post (image)
{
"extension": {},
"body": "드디어 우리 노래 나왔잖아요!!\uD83D\uDC40\uD83E\uDEF6 \n피어나 어때요~~~??\n<w:attachment type=\"photo\" id=\"4-274880925\" />",
"tags": [],
"postId": "1-128752990",
"postType": "NORMAL",
"emotionCount": 30344,
"commentCount": 3812,
"publishedAt": 1698416087773,
"membershipOnly": false,
"availableActions": [
"WRITE_COMMENT",
"BOOKMARK",
"WRITE_EMOTION",
"READ_POST"
],
"sectionType": "ARTIST",
"hideFromArtist": false,
"plainBody": "드디어 우리 노래 나왔잖아요!!\uD83D\uDC40\uD83E\uDEF6 \n피어나 어때요~~~??",
"attachment": {
"photo": {
"4-274880925": {
"url": "https://phinf.wevpstatic.net/MjAyMzEwMjdfMTMy/MDAxNjk4NDE2MDg3NDg0.uCJ26Yp7LLUBOJkCzex48s0BeG9Z8A4lgXZwWAqw64gg.c2KZ81NcEcFF4oJmGqpcuSH7MGAwwq9YkVIq3feLq04g.JPEG/image.jpg",
"width": 2316,
"height": 3088,
"photoId": "4-274880925"
}
}
},
"shareUrl": "https://weverse.io/lesserafim/artist/1-128752990",
"bookmarked": false,
"viewerEmotionId": "3-1441778405",
"locked": false,
"hasProduct": false,
"author": {
"memberId": "b60d95bc3b71f4d97b28ac1b971cc641",
"communityId": 47,
"joined": true,
"profileName": "쥬하\uD83C\uDF3F",
"profileImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjQ2/MDAxNjk3NTU0OTg0MTcz.Ox-UsVCCMxdDfO5eTlOtS3rXjqoY0Rd1RxdN3rt3iTkg.PSneXamakKLjQuB8JR5Fs1R0WxUl5iXsGBS0V_6pzowg.JPEG/c88356d1-8f41-46ed-9f06-2d3810713011.jpeg",
"memberJoinStatus": "JOIN_COMMUNITY",
"profileType": "ARTIST",
"artistOfficialProfile": {
"officialImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjEz/MDAxNjk3NTU0OTc3NjAw.D9zosDQRuK28hOBJsF0JzIeXRn3rkSsXXKa2Japn4XEg.2dYdFv1-y1kjxEqXYtPunIzzGPl7syCSxcbN-Ni0mT4g.JPEG/6faffa29-8c53-4211-b025-69ca813c9ad3.jpeg",
"officialName": "KAZUHA",
"birthday": {
"date": 1060387200000,
"dateString": "2003.08.09",
"timezone": "Asia/Seoul",
"bday": false
}
},
"hasOfficialMark": true,
"profileSpaceStatus": "ACCESSIBLE",
"myProfile": false
},
"community": {
"communityId": 47,
"communityName": "LE SSERAFIM",
"logoImage": "https://phinf.wevpstatic.net/MjAyMzEwMTJfMTI4/MDAxNjk3MDc5MTA3Njg4.tWmeZYTw5wXcgDYW1CdqRI9Sn81B7Fd2V2CF7Qt7bXkg.hOVkhjZpWx8q4YS0LHinzGDsaptQFkM1CvgKEu9aLJgg.PNG/6a41660d-68a1-40a7-8e90-0687914cbc77.png",
"artistCode": "LESSERAFIM",
"homeGradationColor": {
"upLeftColorCode": "#7887c3",
"upRightColorCode": "#455698",
"downLeftColorCode": "#c0cbf5",
"downRightColorCode": "#8e9edf",
"menuTabLeftColorCode": "#465388",
"menuTabRightColorCode": "#1b2e79"
}
}
}
Post (video)
{
"extension": {},
"body": "너무 귀엽죠??\uD83E\uDEF6\n오늘부터 잘 부탁해요~~\uD83D\uDE09 <w:attachment type=\"video\" id=\"4-864785\" />",
"attachment": {
"video": {
"4-864785": {
"videoId": "4-864785",
"uploadInfo": {
"width": 720,
"videoId": "FC3D5A7233E610269DD0BB16F34AC9175D92",
"playTime": 3,
"height": 720,
"imageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMDVfMjU5/MDAxNjk2NDg5MTg0NTg3.MqPrC4culB2vLsB1wjdKLSgzHuIrwNyfbZ2NFVxIXZAg.q5HU37Askl9whC3wZ9lMOymXanLgm68UypqZRYKH8Kgg.JPEG/bf494c1d-634c-11ee-9671-80615f0bcd6e_01.jpg",
"uploadInfoVersion": 1
}
}
}
},
"emotionCount": 28853,
"commentCount": 2379,
"postId": "2-127465849",
"tags": [],
"publishedAt": 1696489184386,
"sectionType": "ARTIST",
"membershipOnly": false,
"availableActions": [
"BOOKMARK",
"WRITE_COMMENT",
"WRITE_EMOTION",
"READ_POST"
],
"hideFromArtist": false,
"postType": "NORMAL",
"plainBody": "너무 귀엽죠??\uD83E\uDEF6\n오늘부터 잘 부탁해요~~\uD83D\uDE09",
"shareUrl": "https://weverse.io/lesserafim/artist/2-127465849",
"bookmarked": false,
"locked": false,
"hasProduct": false,
"author": {
"memberId": "b60d95bc3b71f4d97b28ac1b971cc641",
"communityId": 47,
"joined": true,
"profileName": "쥬하\uD83C\uDF3F",
"profileImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjQ2/MDAxNjk3NTU0OTg0MTcz.Ox-UsVCCMxdDfO5eTlOtS3rXjqoY0Rd1RxdN3rt3iTkg.PSneXamakKLjQuB8JR5Fs1R0WxUl5iXsGBS0V_6pzowg.JPEG/c88356d1-8f41-46ed-9f06-2d3810713011.jpeg",
"memberJoinStatus": "JOIN_COMMUNITY",
"profileType": "ARTIST",
"artistOfficialProfile": {
"officialImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjEz/MDAxNjk3NTU0OTc3NjAw.D9zosDQRuK28hOBJsF0JzIeXRn3rkSsXXKa2Japn4XEg.2dYdFv1-y1kjxEqXYtPunIzzGPl7syCSxcbN-Ni0mT4g.JPEG/6faffa29-8c53-4211-b025-69ca813c9ad3.jpeg",
"officialName": "KAZUHA",
"birthday": {
"date": 1060387200000,
"dateString": "2003.08.09",
"timezone": "Asia/Seoul",
"bday": false
}
},
"hasOfficialMark": true,
"profileSpaceStatus": "ACCESSIBLE",
"myProfile": false
},
"community": {
"communityId": 47,
"communityName": "LE SSERAFIM",
"logoImage": "https://phinf.wevpstatic.net/MjAyMzEwMTJfMTI4/MDAxNjk3MDc5MTA3Njg4.tWmeZYTw5wXcgDYW1CdqRI9Sn81B7Fd2V2CF7Qt7bXkg.hOVkhjZpWx8q4YS0LHinzGDsaptQFkM1CvgKEu9aLJgg.PNG/6a41660d-68a1-40a7-8e90-0687914cbc77.png",
"artistCode": "LESSERAFIM",
"homeGradationColor": {
"upLeftColorCode": "#7887c3",
"upRightColorCode": "#455698",
"downLeftColorCode": "#c0cbf5",
"downRightColorCode": "#8e9edf",
"menuTabLeftColorCode": "#465388",
"menuTabRightColorCode": "#1b2e79"
}
}
}
Post (text)
{
"extension": {},
"tags": [],
"publishedAt": 1696763700370,
"emotionCount": 27997,
"commentCount": 25772,
"body": "피어나 뭐해요\uD83E\uDD79\uD83E\uDEF6",
"availableActions": [
"READ_POST",
"BOOKMARK",
"WRITE_COMMENT",
"WRITE_EMOTION"
],
"membershipOnly": false,
"sectionType": "ARTIST",
"plainBody": "피어나 뭐해요\uD83E\uDD79\uD83E\uDEF6",
"postId": "3-135964325",
"hideFromArtist": false,
"attachment": {},
"postType": "NORMAL",
"shareUrl": "https://weverse.io/lesserafim/artist/3-135964325",
"bookmarked": false,
"locked": false,
"hasProduct": false,
"author": {
"memberId": "b60d95bc3b71f4d97b28ac1b971cc641",
"communityId": 47,
"joined": true,
"profileName": "쥬하\uD83C\uDF3F",
"profileImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjQ2/MDAxNjk3NTU0OTg0MTcz.Ox-UsVCCMxdDfO5eTlOtS3rXjqoY0Rd1RxdN3rt3iTkg.PSneXamakKLjQuB8JR5Fs1R0WxUl5iXsGBS0V_6pzowg.JPEG/c88356d1-8f41-46ed-9f06-2d3810713011.jpeg",
"memberJoinStatus": "JOIN_COMMUNITY",
"profileType": "ARTIST",
"artistOfficialProfile": {
"officialImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjEz/MDAxNjk3NTU0OTc3NjAw.D9zosDQRuK28hOBJsF0JzIeXRn3rkSsXXKa2Japn4XEg.2dYdFv1-y1kjxEqXYtPunIzzGPl7syCSxcbN-Ni0mT4g.JPEG/6faffa29-8c53-4211-b025-69ca813c9ad3.jpeg",
"officialName": "KAZUHA",
"birthday": {
"date": 1060387200000,
"dateString": "2003.08.09",
"timezone": "Asia/Seoul",
"bday": false
}
},
"hasOfficialMark": true,
"profileSpaceStatus": "ACCESSIBLE",
"myProfile": false
},
"artistReactions": [
{
"reactionType": "comment",
"reactionCount": 19,
"member": {
"memberId": "b60d95bc3b71f4d97b28ac1b971cc641",
"communityId": 47,
"joined": true,
"profileName": "쥬하\uD83C\uDF3F",
"profileImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjQ2/MDAxNjk3NTU0OTg0MTcz.Ox-UsVCCMxdDfO5eTlOtS3rXjqoY0Rd1RxdN3rt3iTkg.PSneXamakKLjQuB8JR5Fs1R0WxUl5iXsGBS0V_6pzowg.JPEG/c88356d1-8f41-46ed-9f06-2d3810713011.jpeg",
"memberJoinStatus": "JOIN_COMMUNITY",
"profileType": "ARTIST",
"artistOfficialProfile": {
"officialImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjEz/MDAxNjk3NTU0OTc3NjAw.D9zosDQRuK28hOBJsF0JzIeXRn3rkSsXXKa2Japn4XEg.2dYdFv1-y1kjxEqXYtPunIzzGPl7syCSxcbN-Ni0mT4g.JPEG/6faffa29-8c53-4211-b025-69ca813c9ad3.jpeg",
"officialName": "KAZUHA",
"birthday": {
"date": 1060387200000,
"dateString": "2003.08.09",
"timezone": "Asia/Seoul",
"bday": false
}
},
"hasOfficialMark": true,
"profileSpaceStatus": "ACCESSIBLE",
"myProfile": false
}
}
],
"community": {
"communityId": 47,
"communityName": "LE SSERAFIM",
"logoImage": "https://phinf.wevpstatic.net/MjAyMzEwMTJfMTI4/MDAxNjk3MDc5MTA3Njg4.tWmeZYTw5wXcgDYW1CdqRI9Sn81B7Fd2V2CF7Qt7bXkg.hOVkhjZpWx8q4YS0LHinzGDsaptQFkM1CvgKEu9aLJgg.PNG/6a41660d-68a1-40a7-8e90-0687914cbc77.png",
"artistCode": "LESSERAFIM",
"homeGradationColor": {
"upLeftColorCode": "#7887c3",
"upRightColorCode": "#455698",
"downLeftColorCode": "#c0cbf5",
"downRightColorCode": "#8e9edf",
"menuTabLeftColorCode": "#465388",
"menuTabRightColorCode": "#1b2e79"
}
}
}
Moment
{
"extension": {
"moment": {
"momentAsJson": "{\"description\":\"\",\"style\":\"\",\"version\":\"1.1.3\",\"clips\":[{\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA5MDlfNTIg\\/MDAxNjYyNjk4MDg5MzI3.2G5v-YnreeHKRK0qU9T_5L0PtxKnjO-UxF5Iir4S2dQg.--uDITfIRkc2SJ5e9uDGB1udmfhRBoXVK13hhXcMT8Qg.JPEG\\/C467F406-294F-4D4B-891F-04FDB7F1E6AB.jpg\",\"events\":[],\"title\":\"\",\"videoId\":\"940264D33932D0CD690A57AA820458275CD7\",\"duration\":10,\"metadata\":{\"texts\":[{\"range\":{\"endTime\":10000,\"startTime\":0},\"text\":\"한국에서 처음으로 보낸 추석\uD83E\uDD0D\\n피어나 여러분 뭐해요~~??\"}]},\"resolution\":{\"width\":1080,\"height\":1920}}],\"tags\":[],\"isReviewedByCustomer\":\"N\",\"title\":\"\",\"meta\":{\"stickerCnt\":{\"giphy\":0},\"mediaSrcCnt\":{\"video\":0,\"image\":1}},\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA5MDlfNTIg\\/MDAxNjYyNjk4MDg5MzI3.2G5v-YnreeHKRK0qU9T_5L0PtxKnjO-UxF5Iir4S2dQg.--uDITfIRkc2SJ5e9uDGB1udmfhRBoXVK13hhXcMT8Qg.JPEG\\/C467F406-294F-4D4B-891F-04FDB7F1E6AB.jpg\",\"duration\":10,\"indexes\":[{\"title\":\"\",\"range\":{\"endTime\":10000,\"startTime\":0},\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA5MDlfNTIg\\/MDAxNjYyNjk4MDg5MzI3.2G5v-YnreeHKRK0qU9T_5L0PtxKnjO-UxF5Iir4S2dQg.--uDITfIRkc2SJ5e9uDGB1udmfhRBoXVK13hhXcMT8Qg.JPEG\\/C467F406-294F-4D4B-891F-04FDB7F1E6AB.jpg\"}],\"resolution\":{\"width\":1080,\"height\":1920}}",
"body": "한국에서 처음으로 보낸 추석\uD83E\uDD0D\n피어나 여러분 뭐해요~~??",
"expireAt": 1662784496362,
"video": {
"videoId": "4-83503",
"uploadInfo": {
"width": 1080,
"videoId": "940264D33932D0CD690A57AA820458275CD7",
"playTime": 10,
"height": 1920,
"imageUrl": "https://phinf.wevpstatic.net/MjAyMjA5MDlfNTIg/MDAxNjYyNjk4MDg5MzI3.2G5v-YnreeHKRK0qU9T_5L0PtxKnjO-UxF5Iir4S2dQg.--uDITfIRkc2SJ5e9uDGB1udmfhRBoXVK13hhXcMT8Qg.JPEG/C467F406-294F-4D4B-891F-04FDB7F1E6AB.jpg",
"uploadInfoVersion": 1
}
}
}
},
"tags": [],
"publishedAt": 1662698096362,
"postType": "MOMENT",
"sectionType": "MOMENT",
"membershipOnly": false,
"body": "한국에서 처음으로 보낸 추석\uD83E\uDD0D\n피어나 여러분 뭐해요~~??",
"hideFromArtist": false,
"emotionCount": 4490,
"commentCount": 315,
"availableActions": ["READ_POST"],
"postId": "3-104040928",
"attachment": {},
"plainBody": "한국에서 처음으로 보낸 추석\uD83E\uDD0D\n피어나 여러분 뭐해요~~??",
"shareUrl": "https://weverse.io/lesserafim/moment/b60d95bc3b71f4d97b28ac1b971cc641/post/3-104040928",
"bookmarked": false,
"locked": false,
"hasProduct": false,
"author": {
"memberId": "b60d95bc3b71f4d97b28ac1b971cc641",
"communityId": 47,
"joined": true,
"profileName": "쥬하\uD83C\uDF3F",
"profileImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjQ2/MDAxNjk3NTU0OTg0MTcz.Ox-UsVCCMxdDfO5eTlOtS3rXjqoY0Rd1RxdN3rt3iTkg.PSneXamakKLjQuB8JR5Fs1R0WxUl5iXsGBS0V_6pzowg.JPEG/c88356d1-8f41-46ed-9f06-2d3810713011.jpeg",
"memberJoinStatus": "JOIN_COMMUNITY",
"profileType": "ARTIST",
"artistOfficialProfile": {
"officialImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjEz/MDAxNjk3NTU0OTc3NjAw.D9zosDQRuK28hOBJsF0JzIeXRn3rkSsXXKa2Japn4XEg.2dYdFv1-y1kjxEqXYtPunIzzGPl7syCSxcbN-Ni0mT4g.JPEG/6faffa29-8c53-4211-b025-69ca813c9ad3.jpeg",
"officialName": "KAZUHA",
"birthday": {
"date": 1060387200000,
"dateString": "2003.08.09",
"timezone": "Asia/Seoul",
"bday": false
}
},
"hasOfficialMark": true,
"profileSpaceStatus": "ACCESSIBLE",
"myProfile": false
},
"authorMomentPosts": {
"paging": {},
"data": [
{
"extension": {
"moment": {
"momentAsJson": "{\"description\":\"\",\"style\":\"\",\"version\":\"1.1.3\",\"clips\":[{\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA5MDlfNTIg\\/MDAxNjYyNjk4MDg5MzI3.2G5v-YnreeHKRK0qU9T_5L0PtxKnjO-UxF5Iir4S2dQg.--uDITfIRkc2SJ5e9uDGB1udmfhRBoXVK13hhXcMT8Qg.JPEG\\/C467F406-294F-4D4B-891F-04FDB7F1E6AB.jpg\",\"events\":[],\"title\":\"\",\"videoId\":\"940264D33932D0CD690A57AA820458275CD7\",\"duration\":10,\"metadata\":{\"texts\":[{\"range\":{\"endTime\":10000,\"startTime\":0},\"text\":\"한국에서 처음으로 보낸 추석\uD83E\uDD0D\\n피어나 여러분 뭐해요~~??\"}]},\"resolution\":{\"width\":1080,\"height\":1920}}],\"tags\":[],\"isReviewedByCustomer\":\"N\",\"title\":\"\",\"meta\":{\"stickerCnt\":{\"giphy\":0},\"mediaSrcCnt\":{\"video\":0,\"image\":1}},\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA5MDlfNTIg\\/MDAxNjYyNjk4MDg5MzI3.2G5v-YnreeHKRK0qU9T_5L0PtxKnjO-UxF5Iir4S2dQg.--uDITfIRkc2SJ5e9uDGB1udmfhRBoXVK13hhXcMT8Qg.JPEG\\/C467F406-294F-4D4B-891F-04FDB7F1E6AB.jpg\",\"duration\":10,\"indexes\":[{\"title\":\"\",\"range\":{\"endTime\":10000,\"startTime\":0},\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA5MDlfNTIg\\/MDAxNjYyNjk4MDg5MzI3.2G5v-YnreeHKRK0qU9T_5L0PtxKnjO-UxF5Iir4S2dQg.--uDITfIRkc2SJ5e9uDGB1udmfhRBoXVK13hhXcMT8Qg.JPEG\\/C467F406-294F-4D4B-891F-04FDB7F1E6AB.jpg\"}],\"resolution\":{\"width\":1080,\"height\":1920}}",
"body": "한국에서 처음으로 보낸 추석\uD83E\uDD0D\n피어나 여러분 뭐해요~~??",
"expireAt": 1662784496362,
"video": {
"videoId": "4-83503",
"uploadInfo": {
"width": 1080,
"videoId": "940264D33932D0CD690A57AA820458275CD7",
"playTime": 10,
"height": 1920,
"imageUrl": "https://phinf.wevpstatic.net/MjAyMjA5MDlfNTIg/MDAxNjYyNjk4MDg5MzI3.2G5v-YnreeHKRK0qU9T_5L0PtxKnjO-UxF5Iir4S2dQg.--uDITfIRkc2SJ5e9uDGB1udmfhRBoXVK13hhXcMT8Qg.JPEG/C467F406-294F-4D4B-891F-04FDB7F1E6AB.jpg",
"uploadInfoVersion": 1
}
}
}
},
"tags": [],
"publishedAt": 1662698096362,
"postType": "MOMENT",
"sectionType": "MOMENT",
"membershipOnly": false,
"hideFromArtist": false,
"emotionCount": 4490,
"commentCount": 315,
"availableActions": ["READ_POST"],
"postId": "3-104040928",
"summary": {
"videoCount": 1,
"thumbnails": [
{
"type": "VIDEO",
"url": "https://phinf.wevpstatic.net/MjAyMjA5MDlfNTIg/MDAxNjYyNjk4MDg5MzI3.2G5v-YnreeHKRK0qU9T_5L0PtxKnjO-UxF5Iir4S2dQg.--uDITfIRkc2SJ5e9uDGB1udmfhRBoXVK13hhXcMT8Qg.JPEG/C467F406-294F-4D4B-891F-04FDB7F1E6AB.jpg",
"playTime": 10,
"cVideoId": "4-83503"
}
],
"photoCount": 0,
"snippets": []
},
"plainBody": "한국에서 처음으로 보낸 추석\uD83E\uDD0D\n피어나 여러분 뭐해요~~??",
"shareUrl": "https://weverse.io/lesserafim/moment/b60d95bc3b71f4d97b28ac1b971cc641/post/3-104040928",
"bookmarked": false,
"locked": false,
"hasProduct": false,
"author": {
"memberId": "b60d95bc3b71f4d97b28ac1b971cc641",
"communityId": 47,
"joined": true,
"profileName": "쥬하\uD83C\uDF3F",
"profileImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjQ2/MDAxNjk3NTU0OTg0MTcz.Ox-UsVCCMxdDfO5eTlOtS3rXjqoY0Rd1RxdN3rt3iTkg.PSneXamakKLjQuB8JR5Fs1R0WxUl5iXsGBS0V_6pzowg.JPEG/c88356d1-8f41-46ed-9f06-2d3810713011.jpeg",
"memberJoinStatus": "JOIN_COMMUNITY",
"profileType": "ARTIST",
"artistOfficialProfile": {
"officialImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjEz/MDAxNjk3NTU0OTc3NjAw.D9zosDQRuK28hOBJsF0JzIeXRn3rkSsXXKa2Japn4XEg.2dYdFv1-y1kjxEqXYtPunIzzGPl7syCSxcbN-Ni0mT4g.JPEG/6faffa29-8c53-4211-b025-69ca813c9ad3.jpeg",
"officialName": "KAZUHA",
"birthday": {
"date": 1060387200000,
"dateString": "2003.08.09",
"timezone": "Asia/Seoul",
"bday": false
}
},
"hasOfficialMark": true,
"profileSpaceStatus": "ACCESSIBLE",
"myProfile": false
},
"community": {
"communityId": 47,
"communityName": "LE SSERAFIM",
"logoImage": "https://phinf.wevpstatic.net/MjAyMzEwMTJfMTI4/MDAxNjk3MDc5MTA3Njg4.tWmeZYTw5wXcgDYW1CdqRI9Sn81B7Fd2V2CF7Qt7bXkg.hOVkhjZpWx8q4YS0LHinzGDsaptQFkM1CvgKEu9aLJgg.PNG/6a41660d-68a1-40a7-8e90-0687914cbc77.png",
"artistCode": "LESSERAFIM"
}
},
{
"extension": {
"moment": {
"momentAsJson": "{\"description\":\"\",\"style\":\"\",\"version\":\"1.1.3\",\"clips\":[{\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA3MjVfNzgg\\/MDAxNjU4NzM4OTg1MDUx.a5ffKA4rCn8iauE6VSqaC5o5QK0EoNTInA4KJKrcj-cg.S7nApdPG43lKfNGUEXA5kmPLQ-RN86D3J3u3mekW-u4g.JPEG\\/E48A02C9-A2FE-40E1-9C05-5AC5441A4EF7.jpg\",\"events\":[],\"title\":\"\",\"videoId\":\"45D6EC7EF41379A783252970FFAC04104648\",\"duration\":10,\"metadata\":{\"texts\":[{\"range\":{\"endTime\":10000,\"startTime\":0},\"text\":\"진짜 오늘 날씨 좋다~~~\uD83E\uDD0D\\n\\n저녁에도 밝은 여름 최고\uD83E\uDD1F\\n\\n좋은 하루 보내세요~~!!\"}]},\"resolution\":{\"width\":1080,\"height\":1920}}],\"tags\":[],\"isReviewedByCustomer\":\"N\",\"title\":\"\",\"meta\":{\"stickerCnt\":{\"giphy\":1},\"mediaSrcCnt\":{\"video\":0,\"image\":1}},\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA3MjVfNzgg\\/MDAxNjU4NzM4OTg1MDUx.a5ffKA4rCn8iauE6VSqaC5o5QK0EoNTInA4KJKrcj-cg.S7nApdPG43lKfNGUEXA5kmPLQ-RN86D3J3u3mekW-u4g.JPEG\\/E48A02C9-A2FE-40E1-9C05-5AC5441A4EF7.jpg\",\"duration\":10,\"indexes\":[{\"title\":\"\",\"range\":{\"endTime\":10000,\"startTime\":0},\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA3MjVfNzgg\\/MDAxNjU4NzM4OTg1MDUx.a5ffKA4rCn8iauE6VSqaC5o5QK0EoNTInA4KJKrcj-cg.S7nApdPG43lKfNGUEXA5kmPLQ-RN86D3J3u3mekW-u4g.JPEG\\/E48A02C9-A2FE-40E1-9C05-5AC5441A4EF7.jpg\"}],\"resolution\":{\"width\":1080,\"height\":1920}}",
"body": "진짜 오늘 날씨 좋다~~~\uD83E\uDD0D\n\n저녁에도 밝은 여름 최고\uD83E\uDD1F\n\n좋은 하루 보내세요~~!!",
"expireAt": 1658825391456,
"video": {
"videoId": "3-15501",
"uploadInfo": {
"width": 1080,
"videoId": "45D6EC7EF41379A783252970FFAC04104648",
"playTime": 10,
"height": 1920,
"imageUrl": "https://phinf.wevpstatic.net/MjAyMjA3MjVfNzgg/MDAxNjU4NzM4OTg1MDUx.a5ffKA4rCn8iauE6VSqaC5o5QK0EoNTInA4KJKrcj-cg.S7nApdPG43lKfNGUEXA5kmPLQ-RN86D3J3u3mekW-u4g.JPEG/E48A02C9-A2FE-40E1-9C05-5AC5441A4EF7.jpg",
"uploadInfoVersion": 1
}
}
}
},
"tags": [],
"publishedAt": 1658738991456,
"postType": "MOMENT",
"sectionType": "MOMENT",
"membershipOnly": false,
"hideFromArtist": false,
"emotionCount": 6976,
"commentCount": 468,
"availableActions": ["READ_POST"],
"postId": "4-101115099",
"summary": {
"videoCount": 1,
"thumbnails": [
{
"type": "VIDEO",
"url": "https://phinf.wevpstatic.net/MjAyMjA3MjVfNzgg/MDAxNjU4NzM4OTg1MDUx.a5ffKA4rCn8iauE6VSqaC5o5QK0EoNTInA4KJKrcj-cg.S7nApdPG43lKfNGUEXA5kmPLQ-RN86D3J3u3mekW-u4g.JPEG/E48A02C9-A2FE-40E1-9C05-5AC5441A4EF7.jpg",
"playTime": 10,
"cVideoId": "3-15501"
}
],
"photoCount": 0,
"snippets": []
},
"plainBody": "진짜 오늘 날씨 좋다~~~\uD83E\uDD0D\n저녁에도 밝은 여름 최고\uD83E\uDD1F\n좋은 하루 보내세요~~!!",
"shareUrl": "https://weverse.io/lesserafim/moment/b60d95bc3b71f4d97b28ac1b971cc641/post/4-101115099",
"bookmarked": false,
"locked": false,
"hasProduct": false,
"author": {
"memberId": "b60d95bc3b71f4d97b28ac1b971cc641",
"communityId": 47,
"joined": true,
"profileName": "쥬하\uD83C\uDF3F",
"profileImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjQ2/MDAxNjk3NTU0OTg0MTcz.Ox-UsVCCMxdDfO5eTlOtS3rXjqoY0Rd1RxdN3rt3iTkg.PSneXamakKLjQuB8JR5Fs1R0WxUl5iXsGBS0V_6pzowg.JPEG/c88356d1-8f41-46ed-9f06-2d3810713011.jpeg",
"memberJoinStatus": "JOIN_COMMUNITY",
"profileType": "ARTIST",
"artistOfficialProfile": {
"officialImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjEz/MDAxNjk3NTU0OTc3NjAw.D9zosDQRuK28hOBJsF0JzIeXRn3rkSsXXKa2Japn4XEg.2dYdFv1-y1kjxEqXYtPunIzzGPl7syCSxcbN-Ni0mT4g.JPEG/6faffa29-8c53-4211-b025-69ca813c9ad3.jpeg",
"officialName": "KAZUHA",
"birthday": {
"date": 1060387200000,
"dateString": "2003.08.09",
"timezone": "Asia/Seoul",
"bday": false
}
},
"hasOfficialMark": true,
"profileSpaceStatus": "ACCESSIBLE",
"myProfile": false
},
"community": {
"communityId": 47,
"communityName": "LE SSERAFIM",
"logoImage": "https://phinf.wevpstatic.net/MjAyMzEwMTJfMTI4/MDAxNjk3MDc5MTA3Njg4.tWmeZYTw5wXcgDYW1CdqRI9Sn81B7Fd2V2CF7Qt7bXkg.hOVkhjZpWx8q4YS0LHinzGDsaptQFkM1CvgKEu9aLJgg.PNG/6a41660d-68a1-40a7-8e90-0687914cbc77.png",
"artistCode": "LESSERAFIM"
}
},
{
"extension": {
"moment": {
"momentAsJson": "{\"description\":\"\",\"style\":\"\",\"version\":\"1.1.3\",\"clips\":[{\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA3MTlfMjg1\\/MDAxNjU4MTkwNjA1ODg3.qtMr3ltfmVDFt-XLFzXLLIjrJXu_On12_zsNPi6GYi4g.uAvNvpkpOJS6wwo1ckrIJeHSEM-shrNOMWW5WopHqXgg.JPEG\\/95CE3178-E10B-4298-B877-396268E6798F.jpg\",\"events\":[],\"title\":\"\",\"videoId\":\"233B24713027EC3B674EDDE9FB61461CF404\",\"duration\":10,\"metadata\":{\"texts\":[{\"range\":{\"endTime\":10000,\"startTime\":0},\"text\":\"우와우\\n이런 것 있었는지 몰랐어!!ㅋㅋㅋㅋ \\n더 생기면 좋겠다\uD83E\uDEF6ㅋㅋㅋ\"}]},\"resolution\":{\"width\":1080,\"height\":1920}}],\"tags\":[],\"isReviewedByCustomer\":\"N\",\"title\":\"\",\"meta\":{\"stickerCnt\":{\"giphy\":3},\"mediaSrcCnt\":{\"video\":0,\"image\":1}},\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA3MTlfMjg1\\/MDAxNjU4MTkwNjA1ODg3.qtMr3ltfmVDFt-XLFzXLLIjrJXu_On12_zsNPi6GYi4g.uAvNvpkpOJS6wwo1ckrIJeHSEM-shrNOMWW5WopHqXgg.JPEG\\/95CE3178-E10B-4298-B877-396268E6798F.jpg\",\"duration\":10,\"indexes\":[{\"title\":\"\",\"range\":{\"endTime\":10000,\"startTime\":0},\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA3MTlfMjg1\\/MDAxNjU4MTkwNjA1ODg3.qtMr3ltfmVDFt-XLFzXLLIjrJXu_On12_zsNPi6GYi4g.uAvNvpkpOJS6wwo1ckrIJeHSEM-shrNOMWW5WopHqXgg.JPEG\\/95CE3178-E10B-4298-B877-396268E6798F.jpg\"}],\"resolution\":{\"width\":1080,\"height\":1920}}",
"body": "우와우\n이런 것 있었는지 몰랐어!!ㅋㅋㅋㅋ \n더 생기면 좋겠다\uD83E\uDEF6ㅋㅋㅋ",
"expireAt": 1658277011392,
"video": {
"videoId": "3-4450",
"uploadInfo": {
"width": 1080,
"videoId": "233B24713027EC3B674EDDE9FB61461CF404",
"playTime": 10,
"height": 1920,
"imageUrl": "https://phinf.wevpstatic.net/MjAyMjA3MTlfMjg1/MDAxNjU4MTkwNjA1ODg3.qtMr3ltfmVDFt-XLFzXLLIjrJXu_On12_zsNPi6GYi4g.uAvNvpkpOJS6wwo1ckrIJeHSEM-shrNOMWW5WopHqXgg.JPEG/95CE3178-E10B-4298-B877-396268E6798F.jpg",
"uploadInfoVersion": 1
}
}
}
},
"tags": [],
"publishedAt": 1658190611392,
"postType": "MOMENT",
"sectionType": "MOMENT",
"membershipOnly": false,
"hideFromArtist": false,
"emotionCount": 4157,
"commentCount": 252,
"availableActions": ["READ_POST"],
"postId": "4-100700103",
"summary": {
"videoCount": 1,
"thumbnails": [
{
"type": "VIDEO",
"url": "https://phinf.wevpstatic.net/MjAyMjA3MTlfMjg1/MDAxNjU4MTkwNjA1ODg3.qtMr3ltfmVDFt-XLFzXLLIjrJXu_On12_zsNPi6GYi4g.uAvNvpkpOJS6wwo1ckrIJeHSEM-shrNOMWW5WopHqXgg.JPEG/95CE3178-E10B-4298-B877-396268E6798F.jpg",
"playTime": 10,
"cVideoId": "3-4450"
}
],
"photoCount": 0,
"snippets": []
},
"plainBody": "우와우\n이런 것 있었는지 몰랐어!!ㅋㅋㅋㅋ \n더 생기면 좋겠다\uD83E\uDEF6ㅋㅋㅋ",
"shareUrl": "https://weverse.io/lesserafim/moment/b60d95bc3b71f4d97b28ac1b971cc641/post/4-100700103",
"bookmarked": false,
"locked": false,
"hasProduct": false,
"author": {
"memberId": "b60d95bc3b71f4d97b28ac1b971cc641",
"communityId": 47,
"joined": true,
"profileName": "쥬하\uD83C\uDF3F",
"profileImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjQ2/MDAxNjk3NTU0OTg0MTcz.Ox-UsVCCMxdDfO5eTlOtS3rXjqoY0Rd1RxdN3rt3iTkg.PSneXamakKLjQuB8JR5Fs1R0WxUl5iXsGBS0V_6pzowg.JPEG/c88356d1-8f41-46ed-9f06-2d3810713011.jpeg",
"memberJoinStatus": "JOIN_COMMUNITY",
"profileType": "ARTIST",
"artistOfficialProfile": {
"officialImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjEz/MDAxNjk3NTU0OTc3NjAw.D9zosDQRuK28hOBJsF0JzIeXRn3rkSsXXKa2Japn4XEg.2dYdFv1-y1kjxEqXYtPunIzzGPl7syCSxcbN-Ni0mT4g.JPEG/6faffa29-8c53-4211-b025-69ca813c9ad3.jpeg",
"officialName": "KAZUHA",
"birthday": {
"date": 1060387200000,
"dateString": "2003.08.09",
"timezone": "Asia/Seoul",
"bday": false
}
},
"hasOfficialMark": true,
"profileSpaceStatus": "ACCESSIBLE",
"myProfile": false
},
"community": {
"communityId": 47,
"communityName": "LE SSERAFIM",
"logoImage": "https://phinf.wevpstatic.net/MjAyMzEwMTJfMTI4/MDAxNjk3MDc5MTA3Njg4.tWmeZYTw5wXcgDYW1CdqRI9Sn81B7Fd2V2CF7Qt7bXkg.hOVkhjZpWx8q4YS0LHinzGDsaptQFkM1CvgKEu9aLJgg.PNG/6a41660d-68a1-40a7-8e90-0687914cbc77.png",
"artistCode": "LESSERAFIM"
}
}
]
},
"community": {
"communityId": 47,
"communityName": "LE SSERAFIM",
"logoImage": "https://phinf.wevpstatic.net/MjAyMzEwMTJfMTI4/MDAxNjk3MDc5MTA3Njg4.tWmeZYTw5wXcgDYW1CdqRI9Sn81B7Fd2V2CF7Qt7bXkg.hOVkhjZpWx8q4YS0LHinzGDsaptQFkM1CvgKEu9aLJgg.PNG/6a41660d-68a1-40a7-8e90-0687914cbc77.png",
"artistCode": "LESSERAFIM",
"homeGradationColor": {
"upLeftColorCode": "#7887c3",
"upRightColorCode": "#455698",
"downLeftColorCode": "#c0cbf5",
"downRightColorCode": "#8e9edf",
"menuTabLeftColorCode": "#465388",
"menuTabRightColorCode": "#1b2e79"
}
}
}
For endpoints with pagination support, their response includes a paging.nextParams.after value that can be sent as an after param in the next request.
Some of these values could probably be removed from the final metadata (or the metadata could be created from scratch with cherry picked values), though I'm hesitant to remove too much for versatility with naming/filtering.
I found myself needing to pull an all-nighter, so I went ahead and just implemented most of the TODOs to keep myself busy.
Some of the extractors probably aren't necessary. WeverseMomentsExtractor in particular could probably be merged into WeverseMomentExtractor, and an optional capture group could be used for single Moments instead. I wanted to keep things separate for now as I'm still learning how everything works.
@mikf Feel free to tear this to pieces, I want it to work well.
I'm decently happy with this feature-wise. There are some remaining issues:
- The code could be much DRY-er. I'm currently working on extracting some logic to functions.
- The code should better adhere to current conventions.
- I'm not a fan of how I'm currently handling login, as
self.login()is executed every time an extractor is called fromMessage.Queue. Is this just a consequence of how I've structured the code (extractors deferring to other extractors)?- Somewhat related, I'm seeing a lot of logs about resetting dropped connections when making requests. Do I need to init and reuse a session somewhere to prevent this?
- If possible, for embeds (which are currently just YouTube videos as far as I can tell, I haven't found an embed that isn't a YouTube video yet), I would like to be able to set the directory to
directory_fmt, but allowytdlto set the filename instead of usingfilename_fmt. I havedownload.ytdl.outtmplset tonullin my test config. Is it possible to set download options per extractor (e.g.extractor.weverse.download.ytdl.outtmpl)? Or are there any code changes I can make to achieve this?
So far, I've manually tested (with --config-ignore --config ./config.json --verbose -- <URL>) the following:
| Task | Defaults | skip-embeds = true |
videos = "ytdl" |
videos = false |
|---|---|---|---|---|
| Login (cookies from browser) | ✅ | |||
Login (access-token config set) |
✅ | |||
Login (username and password config set) |
✅ | |||
Artist Post (image) |
✅ | |||
Artist Post (video) |
✅ | ✅ | ✅ | |
Artist Post (text) |
✅ | |||
Fan Post (image) |
✅ | |||
Artist Profile |
✅ | ✅ | ✅ | |
Artists Tab |
✅ | ✅ | ✅ | |
Moment |
✅ | ✅ | ✅ | |
Artist Moments |
✅ | ✅ | ✅ | |
Media (image) |
✅ | |||
Media (video) |
✅ | ✅ | ✅ | |
Media (embed) |
✅ | ✅ | ✅ | ✅ |
Media Tab |
✅ | ✅ | ✅ | ✅ |
Media Category |
✅ | ✅ | ✅ | ✅ |
with this base config.json:
{
"downloader": {
"mtime": false,
"ytdl": {
"module": "yt_dlp",
"outtmpl": null
}
},
"extractor": {
"path-restrict": "windows",
"postprocessors": [
{
"key": "date",
"name": "mtime"
}
],
"weverse": {
"postprocessors": [
{
"async": false,
"command": [
"exiftool",
"-quiet",
"-overwrite_original",
"-preserve",
"-charset",
"utf8",
"-AllComments={shareUrl}/",
"{_path[4:]}"
],
"event": "after",
"name": "exec"
}
],
},
}
}
I don't have any active paid memberships, so I don't know if their API endpoints/responses differ at all. I can look into it if/when these initial extractors are merged. Some VOD content also appears to be completely DRM protected.
I found a new endpoint which was the missing piece to allow all non-paginating extractors to work without authentication. I have now added some more robust tests.
@mikf I believe this is pretty much finished now (pending a review at your convenience, of course).
I attempted to rewrite it in the style of some other extractors such as twitter and bluesky (e.g. using something like posts() instead of items() where possible), but I'm unsure if there are enough similarities between Weverse and those sites for it to be of any use, so I reverted to delegating for all extractors that use pagination.
ytdl support (for non-embed videos) has been removed. Only yt-dlp has Weverse extractors (for Media videos and live streams/VODs, not Post videos), I am not sure if ytdl support adds much value here. ytdl is still used if embeds is enabled.
I also have most if not all possible relevant API responses saved if you'd like them for context (or to create the extractors yourself if you prefer).
