Compare commits

...

50 Commits

Author SHA1 Message Date
Isaac Abadi
51512a60da Fixed docker build issues 2022-05-01 00:22:28 -04:00
GlassedSilver
960c545f37 Dockerfile uses Ubuntu 20.04, fix obtain ffmpeg 2022-05-01 05:14:31 +02:00
Isaac Abadi
5e3eb68b03 Fixed issue where setting sub downloads as 'fresh' was not working properly (#567) 2022-04-30 00:58:12 -04:00
Glassed Silver
4dd3b97515 Merge pull request #566 from GlassedSilver/master
Fixing DNS issues caused by outdated musl version
2022-04-26 06:39:26 +02:00
Glassed Silver
701066eec1 Merge pull request #562 from Tzahi12345/GlassedSilver-add-security-policy
Added Security Policy
2022-04-26 06:39:16 +02:00
GlassedSilver
7f61ccb5f5 Use fixed version of musl to fix DNS errors 2022-04-26 04:46:05 +02:00
Glassed Silver
4f227ca442 Delete extensions.json 2022-04-26 04:28:47 +02:00
Glassed Silver
666bd2057d Merge branch 'Tzahi12345:master' into master 2022-04-26 04:25:50 +02:00
Isaac Abadi
37c858f950 Revert "Updated ffmpeg link in docker-build.sh to use release builds"
This reverts commit 768ec59f30.
2022-04-24 06:16:43 -04:00
Isaac Abadi
ebb7f6a2b0 Revert "Fixed mangled ffmpeg link"
This reverts commit 48e46db071.
2022-04-24 06:16:02 -04:00
Isaac Abadi
48e46db071 Fixed mangled ffmpeg link 2022-04-24 05:51:04 -04:00
Isaac Abadi
768ec59f30 Updated ffmpeg link in docker-build.sh to use release builds 2022-04-24 05:49:09 -04:00
Glassed Silver
aa8f602856 Added Security Policy 2022-04-24 11:12:22 +02:00
Isaac Abadi
d5c1361e64 Fixed issue where roles were not properly initialized 2022-04-24 05:02:02 -04:00
Isaac Abadi
901a96aada Fixed issue where use_local_db may be out of sync when first starting youtubedl-material 2022-04-24 05:01:45 -04:00
Isaac Abadi
21507ee36d Updated methodology of calculating download progress to rely on fs.readdir instead of glob 2022-04-24 04:15:38 -04:00
Isaac Abadi
0585943d67 Fixed bug where task time was not properly set with values of 0
Fixed issue where time field was not properly populated in the schedule dialog
2022-04-24 04:10:27 -04:00
Isaac Abadi
0bc2193f25 Updated downloadFile API request 2022-04-23 21:41:39 -04:00
Isaac Abadi
f3398fce1a Merge branch 'master' of https://github.com/Tzahi12345/YoutubeDL-Material 2022-04-23 21:20:51 -04:00
Glassed Silver
60e8973f52 Merge branch 'Tzahi12345:master' into master 2022-04-24 00:40:58 +02:00
Isaac Abadi
d94857b0a5 Rolled back passport update 2022-04-23 18:14:44 -04:00
Glassed Silver
5fda56d7af Merge pull request #560 from Tzahi12345/revert-556-dependabot/npm_and_yarn/async-2.6.4
Revert "Bump async from 2.6.3 to 2.6.4"
2022-04-23 23:53:29 +02:00
Glassed Silver
abb80b33f3 Revert "Bump async from 2.6.3 to 2.6.4" 2022-04-23 23:53:15 +02:00
Glassed Silver
9977340161 Merge pull request #558 from Tzahi12345/angular-13-update
Angular/dependencies updates
2022-04-23 08:44:27 +02:00
Glassed Silver
8ded160775 Merge branch 'master' into angular-13-update 2022-04-23 08:43:01 +02:00
Glassed Silver
2ee64c7a65 Merge pull request #515 from depuits/master
Added deleteAllFiles api endpoint
2022-04-23 08:27:54 +02:00
Glassed Silver
2ec7efa1ac Merge pull request #555 from Tzahi12345/dependabot/npm_and_yarn/backend/async-3.2.2
Bump async from 3.2.0 to 3.2.2 in /backend
2022-04-23 08:06:54 +02:00
Glassed Silver
4d51384ce6 Merge pull request #556 from Tzahi12345/dependabot/npm_and_yarn/async-2.6.4
Bump async from 2.6.3 to 2.6.4
2022-04-23 08:06:30 +02:00
Isaac Abadi
aa616af118 Fixed issue where navigating from one sub to another didn't cause the new one to load
Fixed subscription downloading as zip

Minor code cleanuip
2022-04-22 23:09:06 -04:00
Isaac Abadi
feebe3e2ba Fixed accidental reversion of styles.scss to much older version 2022-04-22 22:49:05 -04:00
Isaac Abadi
02e90fe818 Fixed issue where icons would not render properly 2022-04-22 22:43:25 -04:00
Isaac Abadi
a4cfafe63c Updated frontend and backend dependencies, fixed some security issues 2022-04-22 22:34:29 -04:00
Isaac Abadi
63e2e6dd3c Fixed build warnings 2022-04-22 22:14:21 -04:00
Isaac Abadi
5a44229e24 Fixed styles.scss 2022-04-22 22:07:50 -04:00
Isaac Abadi
5025b235b7 Updated angular material to v13 2022-04-22 19:03:55 -04:00
Isaac Abadi
5d540fc52a Updated angular to v13 2022-04-22 18:55:11 -04:00
Isaac Abadi
55dfc17d62 Updated ngx-file-drop to support angular v13 2022-04-22 17:51:20 -04:00
Isaac Abadi
2459403b22 Updated angular material 2022-04-22 17:46:45 -04:00
Isaac Abadi
ed5f910c33 Updated angular to v12 2022-04-22 17:40:53 -04:00
Isaac Abadi
468e7153e4 Updated ngx-videogular 2022-04-22 17:28:15 -04:00
dependabot[bot]
1bd713fe17 Bump async from 2.6.3 to 2.6.4
Bumps [async](https://github.com/caolan/async) from 2.6.3 to 2.6.4.
- [Release notes](https://github.com/caolan/async/releases)
- [Changelog](https://github.com/caolan/async/blob/v2.6.4/CHANGELOG.md)
- [Commits](https://github.com/caolan/async/compare/v2.6.3...v2.6.4)

---
updated-dependencies:
- dependency-name: async
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-04-22 19:47:54 +00:00
dependabot[bot]
3df377a260 Bump async from 3.2.0 to 3.2.2 in /backend
Bumps [async](https://github.com/caolan/async) from 3.2.0 to 3.2.2.
- [Release notes](https://github.com/caolan/async/releases)
- [Changelog](https://github.com/caolan/async/blob/master/CHANGELOG.md)
- [Commits](https://github.com/caolan/async/compare/v3.2.0...v3.2.2)

---
updated-dependencies:
- dependency-name: async
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-04-22 19:47:28 +00:00
Glassed Silver
8314ab8fce Merge pull request #554 from Tzahi12345/tasks-and-maintenence-page
Tasks and maintenence page
2022-04-22 21:46:56 +02:00
GlassedSilver
c008171850 add color picker to WS recs 2022-04-21 22:33:01 +02:00
Glassed Silver
0f4f5293de Merge pull request #551 from GlassedSilver/master
Code Cleanup in some places, fix for conversion errors, possibly better webm support, added dependency checks in docker compose file
2022-04-18 10:56:37 +02:00
GlassedSilver
16943847fc fix ffmpeg download with variable 2022-04-18 08:57:21 +02:00
GlassedSilver
f79b254040 using more recent ffmpeg + code cleanup 2022-04-18 07:29:28 +02:00
GlassedSilver
e7989e41f9 Merge branch 'master' of https://github.com/GlassedSilver/YoutubeDL-Material into master 2022-04-18 05:54:09 +02:00
GlassedSilver
a4d421d398 add downloader script for JVS's ffmpeg master blds 2022-04-18 05:54:01 +02:00
Joeri Colman
b46b9ea386 Added deleteAllFiles api endpoint 2022-02-08 10:07:37 +01:00
28 changed files with 5432 additions and 8077 deletions

7
.dockerignore Normal file
View File

@@ -0,0 +1,7 @@
.git
db
appdata
audio
video
subscriptions
users

9
.gitignore vendored
View File

@@ -25,6 +25,7 @@
!.vscode/extensions.json
# misc
/.angular/cache
/.sass-cache
/connect.lock
/coverage
@@ -67,3 +68,11 @@ backend/users/*
backend/appdata/cookies.txt
backend/public
src/assets/i18n/*.json
# User Files
db/
appdata/
audio/
video/
subscriptions/
users/

View File

@@ -1,7 +1,16 @@
FROM alpine:latest as frontend
FROM ubuntu:20.04 AS ffmpeg
RUN apk add --no-cache \
npm
RUN apt-get update && apt-get -y install curl xz-utils
COPY docker-build.sh .
RUN sh ./docker-build.sh
FROM ubuntu:20.04 as frontend
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update
RUN apt-get -y install curl gnupg
RUN curl -sL https://deb.nodesource.com/setup_12.x | bash -
RUN apt-get -y install nodejs
RUN npm install -g @angular/cli
@@ -15,34 +24,35 @@ RUN npm run build
#--------------#
FROM alpine:latest
FROM ubuntu:20.04
ENV UID=1000 \
GID=1000 \
USER=youtube
USER=youtube \
NO_UPDATE_NOTIFIER=true
ENV NO_UPDATE_NOTIFIER=true
RUN groupadd -g $GID $USER && useradd --system -g $USER --uid $UID $USER
RUN addgroup -S $USER -g $GID && adduser -D -S $USER -G $USER -u $UID
RUN apt-get update && apt-get -y install curl
RUN apk add --no-cache \
ffmpeg \
npm \
RUN curl -sL https://deb.nodesource.com/setup_12.x | bash -
RUN apt-get update && apt-get -y install \
nodejs \
python2 \
python3 \
su-exec \
&& apk add --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/ \
atomicparsley
atomicparsley
WORKDIR /app
COPY --from=ffmpeg /usr/local/bin/ffmpeg /usr/local/bin/ffmpeg
COPY --from=ffmpeg /usr/local/bin/ffprobe /usr/local/bin/ffprobe
COPY --chown=$UID:$GID [ "backend/package.json", "backend/package-lock.json", "/app/" ]
ENV PM2_HOME=/app/pm2
RUN npm install pm2 -g
RUN npm install && chown -R $UID:$GID ./
RUN npm install pm2 -g && \
npm install && chown -R $UID:$GID ./
COPY --chown=$UID:$GID --from=frontend [ "/build/backend/public/", "/app/public/" ]
COPY --chown=$UID:$GID [ "/backend/", "/app/" ]
EXPOSE 17442
ENTRYPOINT [ "/app/entrypoint.sh" ]
# ENTRYPOINT [ "/app/entrypoint.sh" ]
CMD [ "pm2-runtime", "pm2.config.js" ]

View File

@@ -16,7 +16,7 @@ paths:
- downloader
summary: Download video file
description: |-
Downloads a video file with the given URL. Will include global args if they exist.
Downloads a file with the given URL. Will include global args if they exist.
HTTP requests will return once the video file download completes. In the future, it will (by default) return once the download starts, and a separate API call will be used for checking the download status.
@@ -41,7 +41,7 @@ paths:
post:
tags:
- downloader
summary: Download video file
summary: Generates arguments used to download file
description: Generates args, used for checking what args would run if you ran downloadFile
operationId: post-generateArgs
requestBody:
@@ -482,6 +482,26 @@ paths:
description: OK
security:
- Auth query parameter: []
/api/deleteAllFiles:
post:
tags:
- files
summary: Delete all downloaded files
operationId: post-api-deleteAllFiles
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/DeleteMp3Mp4Request'
responses:
'200':
description: OK
content:
application/json:
schema:
$ref: '#/components/schemas/DeleteAllFilesResponse'
security:
- Auth query parameter: []
/api/downloadArchive:
post:
tags:
@@ -1765,6 +1785,15 @@ components:
type: boolean
error:
type: string
DeleteAllFilesResponse:
type: object
properties:
file_count:
type: number
description: Number of files found matching search parameters
delete_count:
type: number
description: Number of files removed
DeleteSubscriptionFileRequest:
required:
- file

21
SECURITY.md Normal file
View File

@@ -0,0 +1,21 @@
# Security Policy
## Supported Versions
Currently all work on this project goes into the nightly builds.
4.2's RELEASE build is now quite old and should be considered legacy.
We urge users to use the nightly releases, because the project
constantly sees fixes.
| Version | Supported |
| ------------- | ------------------ |
| 4.2 Nightlies | :white_check_mark: |
| 4.2 Release | :x: |
| < 4.2 | :x: |
## Reporting a Vulnerability
Please file an issue in our GitHub's repo, because this app
isn't meant to be safe to run as public instance yet, but rather as a LAN facing app.
We welcome PRs and help in general in making YTDL-M more secure, but it's not a priority as of now.

View File

@@ -17,7 +17,6 @@
"build": {
"builder": "@angular-devkit/build-angular:browser",
"options": {
"aot": true,
"outputPath": "backend/public",
"index": "src/index.html",
"main": "src/main.ts",
@@ -33,7 +32,17 @@
"styles": [
"src/styles.scss"
],
"scripts": []
"scripts": [],
"vendorChunk": true,
"extractLicenses": false,
"buildOptimizer": false,
"sourceMap": true,
"optimization": false,
"namedChunks": true,
"allowedCommonJsDependencies": [
"rxjs",
"crypto-js"
]
},
"configurations": {
"production": {
@@ -46,7 +55,6 @@
"optimization": true,
"outputHashing": "all",
"namedChunks": false,
"aot": true,
"extractLicenses": true,
"vendorChunk": false,
"buildOptimizer": true,
@@ -60,7 +68,8 @@
"es": {
"localize": ["es"]
}
}
},
"defaultConfiguration": ""
},
"serve": {
"builder": "@angular-devkit/build-angular:dev-server",
@@ -152,16 +161,6 @@
"src/backend"
]
}
},
"lint": {
"builder": "@angular-devkit/build-angular:tslint",
"options": {
"tsConfig": [
"src/tsconfig.app.json",
"src/tsconfig.spec.json"
],
"exclude": []
}
}
}
},
@@ -176,15 +175,6 @@
"protractorConfig": "./protractor.conf.js",
"devServerTarget": "youtube-dl-material:serve"
}
},
"lint": {
"builder": "@angular-devkit/build-angular:tslint",
"options": {
"tsConfig": [
"e2e/tsconfig.e2e.json"
],
"exclude": []
}
}
}
}

View File

@@ -803,7 +803,7 @@ app.post('/api/testConnectionString', optionalJwt, async (req, res) => {
app.post('/api/downloadFile', optionalJwt, async function(req, res) {
req.setTimeout(0); // remove timeout in case of long videos
const url = req.body.url;
const type = req.body.type;
const type = req.body.type ? req.body.type : 'video';
const user_uid = req.isAuthenticated() ? req.user.uid : null;
const options = {
customArgs: req.body.customArgs,
@@ -1434,6 +1434,46 @@ app.post('/api/deleteFile', optionalJwt, async (req, res) => {
res.send(wasDeleted);
});
app.post('/api/deleteAllFiles', optionalJwt, async (req, res) => {
const blacklistMode = false;
const uuid = req.isAuthenticated() ? req.user.uid : null;
let files = null;
let text_search = req.body.text_search;
let file_type_filter = req.body.file_type_filter;
const filter_obj = {user_uid: uuid};
const regex = true;
if (text_search) {
if (regex) {
filter_obj['title'] = {$regex: `.*${text_search}.*`, $options: 'i'};
} else {
filter_obj['$text'] = { $search: utils.createEdgeNGrams(text_search) };
}
}
if (file_type_filter === 'audio_only') filter_obj['isAudio'] = true;
else if (file_type_filter === 'video_only') filter_obj['isAudio'] = false;
files = await db_api.getRecords('files', filter_obj);
let file_count = await db_api.getRecords('files', filter_obj, true);
let delete_count = 0;
for (let i = 0; i < files.length; i++) {
let wasDeleted = false;
wasDeleted = await db_api.deleteFile(files[i].uid, uuid, blacklistMode);
if (wasDeleted) {
delete_count++;
}
}
res.send({
file_count: file_count,
delete_count: delete_count
});
});
app.post('/api/downloadFileFromServer', optionalJwt, async (req, res) => {
let uid = req.body.uid;
let uuid = req.body.uuid;
@@ -1456,18 +1496,14 @@ app.post('/api/downloadFileFromServer', optionalJwt, async (req, res) => {
}
// generate zip
file_path_to_download = await utils.createContainerZipFile(playlist, playlist_files_to_download);
file_path_to_download = await utils.createContainerZipFile(playlist['name'], playlist_files_to_download);
} else if (sub_id && !uid) {
zip_file_generated = true;
const sub_files_to_download = [];
const sub = subscriptions_api.getSubscription(sub_id, uuid);
for (let i = 0; i < sub['videos'].length; i++) {
const sub_file = sub['videos'][i];
sub_files_to_download.push(sub_file);
}
const sub = await db_api.getRecord('subscriptions', {id: sub_id});
const sub_files_to_download = await db_api.getRecords('files', {sub_id: sub_id});
// generate zip
file_path_to_download = await utils.createContainerZipFile(sub, sub_files_to_download);
file_path_to_download = await utils.createContainerZipFile(sub['name'], sub_files_to_download);
} else {
const file_obj = await db_api.getVideo(uid, uuid, sub_id)
file_path_to_download = file_obj.path;

View File

@@ -18,10 +18,19 @@ let JWT_EXPIRATION = null;
let opts = null;
let saltRounds = null;
exports.initialize = function() {
exports.initialize = function () {
/*************************
* Authentication module
************************/
if (db_api.database_initialized) {
setupRoles();
} else {
db_api.database_initialized_bs.subscribe(init => {
if (init) setupRoles();
});
}
saltRounds = 10;
JWT_EXPIRATION = config_api.getConfigItem('ytdl_jwt_expiration');
@@ -49,6 +58,41 @@ exports.initialize = function() {
}));
}
const setupRoles = async () => {
const required_roles = {
admin: {
permissions: [
'filemanager',
'settings',
'subscriptions',
'sharing',
'advanced_download',
'downloads_manager'
]
},
user: {
permissions: [
'filemanager',
'subscriptions',
'sharing'
]
}
}
const role_keys = Object.keys(required_roles);
for (let i = 0; i < role_keys.length; i++) {
const role_key = role_keys[i];
const role_in_db = await db_api.getRecord('roles', {key: role_key});
if (!role_in_db) {
// insert task metadata into table if missing
await db_api.insertRecordIntoTable('roles', {
key: role_key,
permissions: required_roles[role_key]['permissions']
});
}
}
}
exports.passport = require('passport');
exports.passport.serializeUser(function(user, done) {

View File

@@ -85,6 +85,7 @@ exports.initialize = (input_db, input_users_db) => {
}
exports.connectToDB = async (retries = 5, no_fallback = false, custom_connection_string = null) => {
using_local_db = config_api.getConfigItem('ytdl_use_local_db'); // verify
if (using_local_db && !custom_connection_string) return;
const success = await exports._connectToDB(custom_connection_string);
if (success) return true;

View File

@@ -3,7 +3,6 @@ const { uuid } = require('uuidv4');
const path = require('path');
const mergeFiles = require('merge-files');
const NodeID3 = require('node-id3')
const glob = require('glob')
const Mutex = require('async-mutex').Mutex;
const youtubedl = require('youtube-dl');
@@ -583,20 +582,26 @@ async function checkDownloadPercent(download_uid) {
if (!resulting_file_size) return;
let sum_size = 0;
glob(`{${files_to_check_for_progress.join(',')}, }*`, async (err, files) => {
files.forEach(async file => {
try {
const file_stats = fs.statSync(file);
if (file_stats && file_stats.size) {
sum_size += file_stats.size;
}
} catch (e) {
for (let i = 0; i < files_to_check_for_progress.length; i++) {
const file_to_check_for_progress = files_to_check_for_progress[i];
const dir = path.dirname(file_to_check_for_progress);
if (!fs.existsSync(dir)) continue;
fs.readdir(dir, async (err, files) => {
for (let j = 0; j < files.length; j++) {
const file = files[j];
if (!file.includes(path.basename(file_to_check_for_progress))) continue;
try {
const file_stats = fs.statSync(path.join(dir, file));
if (file_stats && file_stats.size) {
sum_size += file_stats.size;
}
} catch (e) {}
}
const percent_complete = (sum_size/resulting_file_size * 100).toFixed(2);
await db_api.updateRecord('download_queue', {uid: download_uid}, {percent_complete: percent_complete});
});
const percent_complete = (sum_size/resulting_file_size * 100).toFixed(2);
await db_api.updateRecord('download_queue', {uid: download_uid}, {percent_complete: percent_complete});
});
}
}
exports.generateNFOFile = (info, output_path) => {

File diff suppressed because it is too large Load Diff

View File

@@ -30,23 +30,21 @@
},
"homepage": "",
"dependencies": {
"archiver": "^3.1.1",
"async": "^3.1.0",
"archiver": "^5.3.1",
"async": "^3.2.3",
"async-mutex": "^0.3.1",
"axios": "^0.21.2",
"bcryptjs": "^2.4.0",
"compression": "^1.7.4",
"config": "^3.2.3",
"exe": "^1.0.2",
"express": "^4.17.1",
"express": "^4.17.3",
"fluent-ffmpeg": "^2.1.2",
"fs-extra": "^9.0.0",
"glob": "^7.1.6",
"jsonwebtoken": "^8.5.1",
"lowdb": "^1.0.0",
"md5": "^2.2.1",
"merge-files": "^0.1.2",
"mocha": "^8.4.0",
"mocha": "^9.2.2",
"moment": "^2.29.2",
"mongodb": "^3.6.9",
"multer": "^1.4.2",
@@ -57,7 +55,7 @@
"passport": "^0.4.1",
"passport-http": "^0.3.0",
"passport-jwt": "^4.0.0",
"passport-ldapauth": "^2.1.4",
"passport-ldapauth": "^3.0.1",
"passport-local": "^1.0.0",
"progress": "^2.0.3",
"ps-node": "^0.1.6",
@@ -66,7 +64,7 @@
"shortid": "^2.2.15",
"unzipper": "^0.10.10",
"uuidv4": "^6.0.6",
"winston": "^3.3.3",
"winston": "^3.7.2",
"xmlbuilder2": "^3.0.2",
"youtube-dl": "^3.0.2"
}

View File

@@ -266,11 +266,17 @@ async function getVideosForSub(sub, user_uid = null) {
}
resolve(false);
} else if (output) {
if (config_api.getConfigItem('ytdl_subscriptions_redownload_fresh_uploads')) {
await setFreshUploads(sub, user_uid);
checkVideosForFreshUploads(sub, user_uid);
}
if (output.length === 0 || (output.length === 1 && output[0] === '')) {
logger.verbose('No additional videos to download for ' + sub.name);
resolve(true);
return;
}
const output_jsons = [];
for (let i = 0; i < output.length; i++) {
let output_json = null;
@@ -294,14 +300,7 @@ async function getVideosForSub(sub, user_uid = null) {
}
resolve(files_to_download);
if (config_api.getConfigItem('ytdl_subscriptions_redownload_fresh_uploads')) {
await setFreshUploads(sub, user_uid);
checkVideosForFreshUploads(sub, user_uid);
}
resolve(true);
}
}
});
}, err => {
logger.error(err);
@@ -473,22 +472,24 @@ async function updateSubscriptionProperty(sub, assignment_obj) {
return true;
}
async function setFreshUploads(sub, user_uid) {
async function setFreshUploads(sub) {
const sub_files = await db_api.getRecords('files', {sub_id: sub.id});
const current_date = new Date().toISOString().split('T')[0].replace(/-/g, '');
sub.videos.forEach(async video => {
if (current_date === video['upload_date'].replace(/-/g, '')) {
sub_files.forEach(async file => {
if (current_date === file['upload_date'].replace(/-/g, '')) {
// set upload as fresh
const video_uid = video['uid'];
await db_api.setVideoProperty(video_uid, {'fresh_upload': true}, user_uid, sub['id']);
const file_uid = file['uid'];
await db_api.setVideoProperty(file_uid, {'fresh_upload': true});
}
});
}
async function checkVideosForFreshUploads(sub, user_uid) {
const sub_files = await db_api.getRecords('files', {sub_id: sub.id});
const current_date = new Date().toISOString().split('T')[0].replace(/-/g, '');
sub.videos.forEach(async video => {
if (video['fresh_upload'] && current_date > video['upload_date'].replace(/-/g, '')) {
await checkVideoIfBetterExists(video, sub, user_uid)
sub_files.forEach(async file => {
if (file['fresh_upload'] && current_date > file['upload_date'].replace(/-/g, '')) {
await checkVideoIfBetterExists(file, sub, user_uid)
}
});
}
@@ -510,13 +511,13 @@ async function checkVideoIfBetterExists(file_obj, sub, user_uid) {
logger.verbose(`Failed to download better version of video ${file_obj['id']}`);
} else if (output) {
logger.verbose(`Successfully upgraded video ${file_obj['id']}'s ${metric_to_compare} from ${file_obj[metric_to_compare]} to ${output[metric_to_compare]}`);
await db_api.setVideoProperty(file_obj['uid'], {[metric_to_compare]: output[metric_to_compare]}, user_uid, sub['id']);
await db_api.setVideoProperty(file_obj['uid'], {[metric_to_compare]: output[metric_to_compare]});
}
});
}
}
});
await db_api.setVideoProperty(file_obj['uid'], {'fresh_upload': false}, user_uid, sub['id']);
await db_api.setVideoProperty(file_obj['uid'], {'fresh_upload': false});
}
// helper functions

View File

@@ -42,9 +42,9 @@ function scheduleJob(task_key, schedule) {
if (schedule['type'] === 'timestamp') {
converted_schedule = new Date(schedule['data']['timestamp']);
} else if (schedule['type'] === 'recurring') {
const dayOfWeek = schedule['data']['dayOfWeek'] ? schedule['data']['dayOfWeek'] : null;
const hour = schedule['data']['hour'] ? schedule['data']['hour'] : null;
const minute = schedule['data']['minute'] ? schedule['data']['minute'] : null;
const dayOfWeek = schedule['data']['dayOfWeek'] != null ? schedule['data']['dayOfWeek'] : null;
const hour = schedule['data']['hour'] != null ? schedule['data']['hour'] : null;
const minute = schedule['data']['minute'] != null ? schedule['data']['minute'] : null;
converted_schedule = new scheduler.RecurrenceRule(null, null, null, dayOfWeek, hour, minute);
} else {
logger.error(`Failed to schedule job '${task_key}' as the type '${schedule['type']}' is invalid.`)

View File

@@ -58,13 +58,13 @@ async function getDownloadedFilesByType(basePath, type, full_metadata = false) {
return files;
}
async function createContainerZipFile(container_obj, container_file_objs) {
async function createContainerZipFile(file_name, container_file_objs) {
const container_files_to_download = [];
for (let i = 0; i < container_file_objs.length; i++) {
const container_file_obj = container_file_objs[i];
container_files_to_download.push(container_file_obj.path);
}
return await createZipFile(path.join('appdata', container_obj.name + '.zip'), container_files_to_download);
return await createZipFile(path.join('appdata', file_name + '.zip'), container_files_to_download);
}
async function createZipFile(zip_file_path, file_paths) {

36
docker-build.sh Normal file
View File

@@ -0,0 +1,36 @@
#!/bin/sh
# THANK YOU TALULAH (https://github.com/nottalulah) for your help in figuring this out
# and also optimizing some code with this commit.
# xoxo :D
# set -xeuo pipefail
case $(uname -m) in
x86_64)
ARCH=amd64;;
aarch64)
ARCH=arm64;;
armhf)
ARCH=armhf;;
armv7)
ARCH=armel;;
armv7l)
ARCH=armel;;
*)
echo "Unsupported architecture: $(uname -m)"
exit 1
esac
echo "Architecture: $ARCH"
curl --connect-timeout 5 \
--max-time 10 \
--retry 5 \
--retry-delay 0 \
--retry-max-time 40 \
"https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-${ARCH}-static.tar.xz" -o ffmpeg.txz
mkdir /tmp/ffmpeg
tar xf ffmpeg.txz -C /tmp/ffmpeg
cp /tmp/ffmpeg/*/ffmpeg /usr/local/bin/ffmpeg
cp /tmp/ffmpeg/*/ffprobe /usr/local/bin/ffprobe
rm -rf /tmp/ffmpeg ffmpeg.txz

View File

@@ -7,6 +7,8 @@ services:
ytdl_use_local_db: 'false'
write_ytdl_config: 'true'
restart: always
depends_on:
- ytdl-mongo-db
volumes:
- ./appdata:/app/appdata
- ./audio:/app/audio
@@ -23,5 +25,6 @@ services:
logging:
driver: "none"
container_name: mongo-db
restart: always
volumes:
- ./db/:/data/db

11845
package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@@ -5,7 +5,7 @@
"scripts": {
"ng": "ng",
"start": "ng serve",
"build": "ng build --prod",
"build": "ng build --configuration production",
"prebuild": "node src/postbuild.mjs",
"heroku-postbuild": "npm install --prefix backend",
"test": "ng test",
@@ -20,44 +20,44 @@
},
"private": true,
"dependencies": {
"@angular-devkit/core": "^11.0.4",
"@angular/animations": "^11.0.4",
"@angular/cdk": "^11.0.2",
"@angular/common": "^11.0.4",
"@angular/compiler": "^11.0.4",
"@angular/core": "^11.0.4",
"@angular/forms": "^11.0.4",
"@angular/localize": "^11.0.4",
"@angular/material": "^11.0.2",
"@angular/platform-browser": "^11.0.4",
"@angular/platform-browser-dynamic": "^11.0.4",
"@angular/router": "^11.0.4",
"@angular-devkit/core": "^13.3.3",
"@angular/animations": "^13.3.4",
"@angular/cdk": "^13.3.4",
"@angular/common": "^13.3.4",
"@angular/compiler": "^13.3.4",
"@angular/core": "^13.3.4",
"@angular/forms": "^13.3.4",
"@angular/localize": "^13.3.4",
"@angular/material": "^13.3.4",
"@angular/platform-browser": "^13.3.4",
"@angular/platform-browser-dynamic": "^13.3.4",
"@angular/router": "^13.3.4",
"@fontsource/material-icons": "^4.5.4",
"@ngneat/content-loader": "^5.0.0",
"@videogular/ngx-videogular": "^2.1.0",
"@videogular/ngx-videogular": "^5.0.1",
"core-js": "^2.4.1",
"crypto-js": "^4.1.1",
"file-saver": "^2.0.2",
"filesize": "^6.1.0",
"fingerprintjs2": "^2.1.0",
"fs-extra": "^10.0.0",
"material-icons": "^0.5.4",
"material-icons": "^1.10.8",
"nan": "^2.14.1",
"ng-lazyload-image": "^7.0.1",
"ngx-avatar": "^4.0.0",
"ngx-file-drop": "^9.0.1",
"ngx-avatars": "^1.3.1",
"ngx-file-drop": "^13.0.0",
"rxjs": "^6.6.3",
"rxjs-compat": "^6.0.0-rc.0",
"tslib": "^2.0.0",
"typescript": "~4.0.5",
"web-animations-js": "^2.3.2",
"typescript": "~4.6.3",
"xliff-to-json": "^1.0.4",
"zone.js": "~0.10.2"
"zone.js": "~0.11.4"
},
"devDependencies": {
"@angular-devkit/build-angular": "^0.1100.4",
"@angular/cli": "^11.0.4",
"@angular/compiler-cli": "^11.0.4",
"@angular/language-service": "^11.0.4",
"@angular-devkit/build-angular": "^13.3.3",
"@angular/cli": "^13.3.3",
"@angular/compiler-cli": "^13.3.4",
"@angular/language-service": "^13.3.4",
"@types/core-js": "^2.5.2",
"@types/file-saver": "^2.0.1",
"@types/jasmine": "~3.6.0",
@@ -75,7 +75,7 @@
"karma-coverage-istanbul-reporter": "~3.0.2",
"karma-jasmine": "~4.0.0",
"karma-jasmine-html-reporter": "^1.5.0",
"openapi-typescript-codegen": "^0.4.11",
"openapi-typescript-codegen": "^0.21.0",
"protractor": "~7.0.0",
"ts-node": "~3.0.4",
"tslint": "~6.1.0"

View File

@@ -1,11 +1,11 @@
/* Coolors Exported Palette - coolors.co/e8aeb7-b8e1ff-a9fff7-94fbab-82aba1 */
/* HSL */
$color1: hsla(351%, 56%, 80%, 1);
$softblue: hsla(205%, 100%, 86%, 1);
$color3: hsla(174%, 100%, 83%, 1);
$color4: hsla(133%, 93%, 78%, 1);
$color5: hsla(165%, 20%, 59%, 1);
$color1: hsla(351, 56%, 80%, 1);
$softblue: hsla(205, 100%, 86%, 1);
$color3: hsla(174, 100%, 83%, 1);
$color4: hsla(133, 93%, 78%, 1);
$color5: hsla(165, 20%, 59%, 1);
/* RGB */
$color1: rgba(232, 174, 183, 1);

View File

@@ -45,8 +45,6 @@ import { VgBufferingModule } from '@videogular/ngx-videogular/buffering';
import { VgOverlayPlayModule } from '@videogular/ngx-videogular/overlay-play';
import { VgCoreModule } from '@videogular/ngx-videogular/core';
import { InputDialogComponent } from './input-dialog/input-dialog.component';
import { LazyLoadImageModule, IsVisibleProps } from 'ng-lazyload-image';
import { audioFilesMouseHovering, videoFilesMouseHovering, audioFilesOpened, videoFilesOpened } from './main/main.component';
import { CreatePlaylistComponent } from './create-playlist/create-playlist.component';
import { SubscriptionsComponent } from './subscriptions/subscriptions.component';
import { SubscribeDialogComponent } from './dialogs/subscribe-dialog/subscribe-dialog.component';
@@ -56,7 +54,7 @@ import { SubscriptionInfoDialogComponent } from './dialogs/subscription-info-dia
import { SettingsComponent } from './settings/settings.component';
import { MatChipsModule } from '@angular/material/chips';
import { NgxFileDropModule } from 'ngx-file-drop';
import { AvatarModule } from 'ngx-avatar';
import { AvatarModule } from 'ngx-avatars';
import { ContentLoaderModule } from '@ngneat/content-loader';
import es from '@angular/common/locales/es';
@@ -94,122 +92,110 @@ import { RestoreDbDialogComponent } from './dialogs/restore-db-dialog/restore-db
registerLocaleData(es, 'es');
export function isVisible({ event, element, scrollContainer, offset }: IsVisibleProps<any>) {
return (element.id === 'video' ? videoFilesMouseHovering || videoFilesOpened : audioFilesMouseHovering || audioFilesOpened);
}
@NgModule({
declarations: [
AppComponent,
MainComponent,
PlayerComponent,
InputDialogComponent,
CreatePlaylistComponent,
SubscriptionsComponent,
SubscribeDialogComponent,
SubscriptionComponent,
SubscriptionFileCardComponent,
SubscriptionInfoDialogComponent,
SettingsComponent,
AboutDialogComponent,
VideoInfoDialogComponent,
ArgModifierDialogComponent,
HighlightPipe,
LinkifyPipe,
UpdaterComponent,
UpdateProgressDialogComponent,
ShareMediaDialogComponent,
LoginComponent,
DownloadsComponent,
UserProfileDialogComponent,
SetDefaultAdminDialogComponent,
ModifyUsersComponent,
AddUserDialogComponent,
ManageUserComponent,
ManageRoleComponent,
CookiesUploaderDialogComponent,
LogsViewerComponent,
ModifyPlaylistComponent,
ConfirmDialogComponent,
UnifiedFileCardComponent,
RecentVideosComponent,
EditSubscriptionDialogComponent,
CustomPlaylistsComponent,
EditCategoryDialogComponent,
TwitchChatComponent,
SeeMoreComponent,
ConcurrentStreamComponent,
SkipAdButtonComponent,
TasksComponent,
UpdateTaskScheduleDialogComponent,
RestoreDbDialogComponent
],
imports: [
CommonModule,
BrowserModule,
BrowserAnimationsModule,
MatNativeDateModule,
MatRadioModule,
FormsModule,
MatInputModule,
MatSelectModule,
ReactiveFormsModule,
HttpClientModule,
MatToolbarModule,
MatCardModule,
MatSnackBarModule,
MatButtonModule,
MatCheckboxModule,
MatSidenavModule,
MatIconModule,
MatListModule,
MatGridListModule,
MatExpansionModule,
MatProgressBarModule,
MatProgressSpinnerModule,
MatButtonToggleModule,
MatRippleModule,
MatMenuModule,
MatDialogModule,
MatSlideToggleModule,
MatAutocompleteModule,
MatTabsModule,
MatTooltipModule,
MatPaginatorModule,
MatSortModule,
MatTableModule,
MatDatepickerModule,
MatChipsModule,
DragDropModule,
ClipboardModule,
TextFieldModule,
NgxFileDropModule,
AvatarModule,
ContentLoaderModule,
VgCoreModule,
VgControlsModule,
VgOverlayPlayModule,
VgBufferingModule,
LazyLoadImageModule.forRoot({ isVisible }),
RouterModule,
AppRoutingModule,
],
entryComponents: [
InputDialogComponent,
CreatePlaylistComponent,
SubscribeDialogComponent,
SubscriptionInfoDialogComponent,
SettingsComponent
],
providers: [
PostsService,
{ provide: HTTP_INTERCEPTORS, useClass: H401Interceptor, multi: true }
],
exports: [
HighlightPipe,
LinkifyPipe
],
bootstrap: [AppComponent]
declarations: [
AppComponent,
MainComponent,
PlayerComponent,
InputDialogComponent,
CreatePlaylistComponent,
SubscriptionsComponent,
SubscribeDialogComponent,
SubscriptionComponent,
SubscriptionFileCardComponent,
SubscriptionInfoDialogComponent,
SettingsComponent,
AboutDialogComponent,
VideoInfoDialogComponent,
ArgModifierDialogComponent,
HighlightPipe,
LinkifyPipe,
UpdaterComponent,
UpdateProgressDialogComponent,
ShareMediaDialogComponent,
LoginComponent,
DownloadsComponent,
UserProfileDialogComponent,
SetDefaultAdminDialogComponent,
ModifyUsersComponent,
AddUserDialogComponent,
ManageUserComponent,
ManageRoleComponent,
CookiesUploaderDialogComponent,
LogsViewerComponent,
ModifyPlaylistComponent,
ConfirmDialogComponent,
UnifiedFileCardComponent,
RecentVideosComponent,
EditSubscriptionDialogComponent,
CustomPlaylistsComponent,
EditCategoryDialogComponent,
TwitchChatComponent,
SeeMoreComponent,
ConcurrentStreamComponent,
SkipAdButtonComponent,
TasksComponent,
UpdateTaskScheduleDialogComponent,
RestoreDbDialogComponent
],
imports: [
CommonModule,
BrowserModule,
BrowserAnimationsModule,
MatNativeDateModule,
MatRadioModule,
FormsModule,
MatInputModule,
MatSelectModule,
ReactiveFormsModule,
HttpClientModule,
MatToolbarModule,
MatCardModule,
MatSnackBarModule,
MatButtonModule,
MatCheckboxModule,
MatSidenavModule,
MatIconModule,
MatListModule,
MatGridListModule,
MatExpansionModule,
MatProgressBarModule,
MatProgressSpinnerModule,
MatButtonToggleModule,
MatRippleModule,
MatMenuModule,
MatDialogModule,
MatSlideToggleModule,
MatAutocompleteModule,
MatTabsModule,
MatTooltipModule,
MatPaginatorModule,
MatSortModule,
MatTableModule,
MatDatepickerModule,
MatChipsModule,
DragDropModule,
ClipboardModule,
TextFieldModule,
NgxFileDropModule,
AvatarModule,
ContentLoaderModule,
VgCoreModule,
VgControlsModule,
VgOverlayPlayModule,
VgBufferingModule,
RouterModule,
AppRoutingModule,
],
providers: [
PostsService,
{ provide: HTTP_INTERCEPTORS, useClass: H401Interceptor, multi: true }
],
exports: [
HighlightPipe,
LinkifyPipe
],
bootstrap: [AppComponent]
})
export class AppModule { }

View File

@@ -1,6 +1,6 @@
<span class="text" [ngStyle]="{'-webkit-line-clamp': !see_more_active ? line_limit : null}" [innerHTML]="text | linkify"></span>
<span>
<a [routerLink]="" (click)="toggleSeeMore()">
<a [routerLink]="[]" (click)="toggleSeeMore()">
<ng-container *ngIf="!see_more_active" i18n="See more">
See more.
</ng-container>

View File

@@ -62,7 +62,7 @@ export class SkipAdButtonComponent implements OnInit {
getVideoIDHashFromURL(video_id) {
if (!video_id) return null;
return CryptoJS.SHA256(video_id).toString(CryptoJS.enc.Hex);;
return CryptoJS.SHA256(video_id).toString(CryptoJS.enc.Hex);
}
getVideoIDFromURL(url) {

View File

@@ -1,6 +1,6 @@
import { Component, Inject, OnInit } from '@angular/core';
import { MatDialogRef, MAT_DIALOG_DATA } from '@angular/material/dialog';
import { Schedule } from 'api-types';
import { Schedule, Task } from 'api-types';
import { PostsService } from 'app/posts.services';
@Component({
@@ -18,7 +18,7 @@ export class UpdateTaskScheduleDialogComponent implements OnInit {
date = null;
today = new Date();
constructor(@Inject(MAT_DIALOG_DATA) public data: any, private dialogRef: MatDialogRef<UpdateTaskScheduleDialogComponent>, private postsService: PostsService) {
constructor(@Inject(MAT_DIALOG_DATA) public data: {task: Task}, private dialogRef: MatDialogRef<UpdateTaskScheduleDialogComponent>, private postsService: PostsService) {
this.processTask(this.data.task);
this.postsService.getTask(this.data.task.key).subscribe(res => {
this.processTask(res['task']);
@@ -28,7 +28,7 @@ export class UpdateTaskScheduleDialogComponent implements OnInit {
ngOnInit(): void {
}
processTask(task) {
processTask(task: Task): void {
if (!task['schedule']) {
this.enabled = false;
return;
@@ -39,7 +39,11 @@ export class UpdateTaskScheduleDialogComponent implements OnInit {
this.recurring = schedule['type'] === Schedule.type.RECURRING;
if (this.recurring) {
this.time = `${schedule['data']['hour']}:${schedule['data']['minute']}`;
const hour = schedule['data']['hour'];
const minute = schedule['data']['minute'];
// add padding 0s if necessary to hours and minutes
this.time = (hour < 10 ? '0' : '') + hour + ':' + (minute < 10 ? '0' : '') + minute;
if (schedule['data']['dayOfWeek']) {
this.days_of_week = schedule['data']['dayOfWeek'];
@@ -75,7 +79,6 @@ export class UpdateTaskScheduleDialogComponent implements OnInit {
}
} else {
this.date.setHours(hours, minutes);
console.log(this.date);
schedule['data'] = {timestamp: this.date.getTime()};
}
this.dialogRef.close(schedule);

View File

@@ -1,7 +1,6 @@
import {Injectable, isDevMode, Inject} from '@angular/core';
import { HttpClient, HttpParams } from '@angular/common/http';
import 'rxjs/add/operator/map';
import 'rxjs/add/operator/map';
import 'rxjs/add/operator/catch';
import 'rxjs/add/observable/throw';
import { THEMES_CONFIG } from '../themes';

View File

@@ -49,8 +49,8 @@ export class SubscriptionComponent implements OnInit, OnDestroy {
constructor(private postsService: PostsService, private route: ActivatedRoute, private router: Router, private dialog: MatDialog) { }
ngOnInit() {
if (this.route.snapshot.paramMap.get('id')) {
this.id = this.route.snapshot.paramMap.get('id');
this.route.params.subscribe(params => {
this.id = params['id'];
this.postsService.service_initialized.subscribe(init => {
if (init) {
@@ -59,7 +59,7 @@ export class SubscriptionComponent implements OnInit, OnDestroy {
this.sub_interval = setInterval(() => this.getSubscription(true), 1000);
}
});
}
});
// set filter property to cached
const cached_filter_property = localStorage.getItem('filter_property');

View File

@@ -20,44 +20,14 @@ import '@angular/localize/init';
/***************************************************************************************************
* BROWSER POLYFILLS
*/
/** IE9, IE10 and IE11 requires all of the following polyfills. **/
// import 'core-js/es6/symbol';
// import 'core-js/es6/object';
// import 'core-js/es6/function';
// import 'core-js/es6/parse-int';
// import 'core-js/es6/parse-float';
// import 'core-js/es6/number';
// import 'core-js/es6/math';
// import 'core-js/es6/string';
// import 'core-js/es6/date';
// import 'core-js/es6/array';
// import 'core-js/es6/regexp';
// import 'core-js/es6/map';
// import 'core-js/es6/weak-map';
// import 'core-js/es6/set';
/** IE10 and IE11 requires the following for NgClass support on SVG elements */
// import 'classlist.js'; // Run `npm install --save classlist.js`.
/** IE10 and IE11 requires the following to support `@angular/animation`. */
// import 'web-animations-js'; // Run `npm install --save web-animations-js`.
/** Evergreen browsers require these. **/
// import 'core-js/es6/reflect';
/** ALL Firefox browsers require the following to support `@angular/animation`. **/
import 'web-animations-js'; // Run `npm install --save web-animations-js`.
*/ // Run `npm install --save web-animations-js`.
/***************************************************************************************************
* Zone JS is required by Angular itself.
*/
import 'zone.js/dist/zone'; // Included with Angular CLI.
import 'zone.js'; // Included with Angular CLI.

View File

@@ -22,7 +22,9 @@ __karma__.loaded = function () {};
// First, initialize the Angular testing environment.
getTestBed().initTestEnvironment(
BrowserDynamicTestingModule,
platformBrowserDynamicTesting()
platformBrowserDynamicTesting(), {
teardown: { destroyAfterEach: false }
}
);
// Then we find all the tests.
const context = require.context('./', true, /\.spec\.ts$/);