mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-11-24 11:09:12 +08:00
Code Issues Packages Projects Releases Wiki Activity
8,795 Commits 32 Branches 296 Tags 147 MiB
b64411bda8
Commit Graph

5792 Commits

Author SHA1 Message Date
famosss
41af5a6ee5
simplify a bit (#3631) 2022-09-08 21:01:21 -07:00
chrislu
64e75a286e simplify a bit 2022-09-08 19:07:57 -07:00
chrislu
d14d029c73 reduce v.dataFileAccessLock lock scope 2022-09-08 19:05:31 -07:00
chrislu
406a80da4b adjust variable names 2022-09-08 18:54:16 -07:00
chrislu
2bfc8970d2 refactor: move ReadNeedleDataInto into volume_read.go 2022-09-08 18:54:02 -07:00
chrislu
9b084d4c88 purge tcp implementation 2022-09-08 18:03:43 -07:00
chrislu
eb7cf3de81 a new test case 2022-09-08 14:04:42 -07:00
chrislu
4260804613 volume.balance: avoid moving out volume with max=1 2022-09-08 14:04:21 -07:00
chrislu
d653c5f811 unused 2022-09-08 12:27:53 -07:00
chrislu
03e22ddb43 rename 2022-09-08 12:06:03 -07:00
Konstantin Lebedev
31d2f77ceb
refactor https://github.com/seaweedfs/seaweedfs/pull/3616 (#3625) 2022-09-07 23:23:33 -07:00
chrislu
7de112943b Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-07 20:14:11 -07:00
chrislu
7e52b1e195 return err for processRangeRequest() 2022-09-07 20:14:09 -07:00
famosss
449582343f
fix:Sometimes a nil pointer exception is thrown (#3618) 2022-09-07 18:57:13 -07:00
Konstantin Lebedev
695e3a909c
avoid data race on currentDiskUsage.maxVolumeCount (#3592) 
https://github.com/seaweedfs/seaweedfs/issues/3590
 2022-09-07 09:51:15 -07:00
Konstantin Lebedev
cca45b02a2
avoid data race on calc freeVolumeSlotCount (#3594) 
https://github.com/seaweedfs/seaweedfs/issues/3593
 2022-09-07 09:50:17 -07:00
famosss
9678fc2106
fix: volume heartbeat processing error (#3616) 2022-09-07 09:48:51 -07:00
famosss
5ff33eb558
fix: volume Deadlock when exception is thrown (#3613) 
* fix: volume Deadlock when exception is thrown

* adjust log level
 2022-09-06 23:56:29 -07:00
Eric Yang
b324a6536c
ADHOC: add read needle meta grpc (#3581) 
* ADHOC: add read needle meta grpc

* add test

* nit

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-09-06 23:51:27 -07:00
chrislu
c07ab9c060 3.26 2022-09-06 08:26:20 -07:00
Konstantin Lebedev
f4f6855c61
avoid SIGSEGV on localFilerSocket (#3609) 
https://github.com/seaweedfs/seaweedfs/issues/3607
 2022-09-06 08:20:23 -07:00
Konstantin Lebedev
3c3682fcce
more log detail for upload err and deleting (#3577) 2022-09-06 08:19:13 -07:00
Konstantin Lebedev
01a25755c5
[filer] masterClient updates volume adjust log level (#3610) 2022-09-06 07:26:59 -07:00
LHHDZ
bc629665de
fix bug due to data racing on VidMap (#3606) 2022-09-05 20:00:16 -07:00
Konstantin Lebedev
9b2b7d4f5a
avoid data race on GoCDKPubSub.topic (#3596) 2022-09-05 07:57:44 -07:00
Konstantin Lebedev
916673ae24
do not print diskInfos with doVolumeCheckDisk in error log (#3598) 2022-09-05 07:49:26 -07:00
Konstantin Lebedev
5307cadb32
[s3] Replace code 500 to 499 for upload with ErrUnexpectedEOF (#3597) 2022-09-05 07:47:43 -07:00
chrislu
5b38f22e6e 3.25 2022-09-04 22:45:55 -07:00
famosss
131d389fc4
adjust log level (#3589) 2022-09-04 22:37:32 -07:00
chrislu
f4abefc109 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-04 22:23:31 -07:00
chrislu
39340f7e42 cloud tier: s3 consume all read response body 
fix https://github.com/seaweedfs/seaweedfs/issues/3584
 2022-09-04 22:23:28 -07:00
chrislu
7c6324b114 adjust log level 2022-09-04 22:21:24 -07:00
gfx
bf844d8e46
Fix crash in arangodb filer when attempting to access a deleted bucket (#3587) 
* Update helpers.go

* Update arangodb_store_bucket.go
 2022-09-04 20:32:10 -07:00
gfx
3621987727
Correctly pass arangodb driver Transaction ID into the context (#3586) 
Update arangodb_store.go
 2022-09-04 19:29:52 -07:00
chrislu
8c5759a06d minor optimization 2022-09-04 18:50:45 -07:00
chrislu
70777d9b9f Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-04 18:26:51 -07:00
chrislu
cb6cf331ca filer.backup and filer.sync: include headers during backup and sync 
fix https://github.com/seaweedfs/seaweedfs/issues/3532
 2022-09-04 18:26:36 -07:00
famosss
7f56a40553
Turn down the 404 log level (#3585) 2022-09-04 18:17:01 -07:00
Brian
a28b668647
Added ability to change replication settings upon volume.tier.move (#3583) 2022-09-04 16:47:21 -07:00
bernardx
228b133afa
new 'concurrency' parameter for filer.sync (#3579) 
Co-authored-by: XIAOYQ <xiaoyq@eudic.net>
 2022-09-02 23:03:23 -07:00
Konstantin Lebedev
853880bd83
[filer] DATA RACE on signal_handling fixed (#3575) 2022-09-02 06:20:19 -07:00
chrislu
a112cbe44c filer: rocksdb store report not found correctly 
fix https://github.com/seaweedfs/seaweedfs/discussions/3555
 2022-09-01 15:42:59 -07:00
chrislu
d1bb23645e Revert "avoid data race on grace.hooks (#3572)" 
This reverts commit c37d6fc01a.
 2022-09-01 13:08:34 -07:00
Konstantin Lebedev
c37d6fc01a
avoid data race on grace.hooks (#3572) 
https://github.com/seaweedfs/seaweedfs/issues/3564
 2022-09-01 10:34:06 -07:00
Konstantin Lebedev
8c3040db81
avoid DATA RACE on S3Options.localFilerSocket (#3571) 
* avoid DATA RACE on S3Options.localFilerSocket
https://github.com/seaweedfs/seaweedfs/issues/3552

* copy localSocket
 2022-09-01 10:33:23 -07:00
Konstantin Lebedev
90d55cd179
avoid data race read fileChunksSize (#3570) 
https://github.com/seaweedfs/seaweedfs/issues/3557
 2022-09-01 10:31:57 -07:00
Konstantin Lebedev
82c2ce74db
more error details in the log (#3568) 
* more error details in the log
https://github.com/seaweedfs/seaweedfs/issues/3567

* format message
 2022-09-01 09:04:15 -07:00
chrislu
10efdc7aab align memory 2022-09-01 09:02:44 -07:00
Konstantin Lebedev
803ca3c958
avoid data race on doSubscribeToOneFiler/ma.filer.UniqueFilerEpoch (#3566) 
https://github.com/seaweedfs/seaweedfs/issues/3565
 2022-09-01 09:01:47 -07:00
chrislu
bcf35876d1 add more logs 2022-08-31 23:16:05 -07:00
dependabot[bot]
97d69d5336
Bump gocloud.dev/pubsub/rabbitpubsub from 0.25.0 to 0.26.0 (#3541) 
* Bump gocloud.dev/pubsub/rabbitpubsub from 0.25.0 to 0.26.0

Bumps [gocloud.dev/pubsub/rabbitpubsub](https://github.com/google/go-cloud) from 0.25.0 to 0.26.0.
- [Release notes](https://github.com/google/go-cloud/releases)
- [Commits](https://github.com/google/go-cloud/compare/v0.25.0...v0.26.0)

---
updated-dependencies:
- dependency-name: gocloud.dev/pubsub/rabbitpubsub
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* update code

* more code fix

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
 2022-08-31 10:16:49 -07:00
Patrick Schmidt
a73e177ecf
Add an End-to-End workflow for FUSE mount (#3562) 
* Add an e2e workflow to test FUSE mount

* Fix deadlocks during concurrent r/w
 2022-08-31 09:27:53 -07:00
chrislu
4a4ef3cc3c upload_content: upload close response.Body 2022-08-31 00:25:28 -07:00
chrislu
3d8b8ffced refactor 2022-08-31 00:24:29 -07:00
chrislu
ef78631a7c just a bit safer 2022-08-31 00:10:17 -07:00
chrislu
9c944377dc minor 2022-08-31 00:09:32 -07:00
chrislu
a54f30b6c6 s3: close response body with copying object 2022-08-31 00:09:23 -07:00
chrislu
7eb121d5ae print process before do the work 2022-08-30 13:37:05 -07:00
Konstantin Lebedev
77098de69a
migrate from boltdb to bbolt (#3554) 2022-08-30 10:12:02 -07:00
chrislu
de5263364b master client: use GetMaster() for latest master address 2022-08-30 09:52:06 -07:00
chrislu
e583dbdd2a more detailed error 2022-08-30 09:43:17 -07:00
chrislu
d81db3c703 s3: fix configuring IAM for the same user 
hi, how can I add bucket permission to a user now?
Previously, if I needed to add permission to an existing credential, I simply repeated the s3.configure command with a different bucket name.
Now I am getting error:
duplicate accessKey[ХХХХ], already configured in user[YYYY]

s3.configure -access_key key -actions Read,Write,List -buckets bucket1 -secret_key secr -user user1
s3.configure -access_key key -actions Read,Write,List -buckets bucket2 -secret_key secr -user user1
 2022-08-30 09:37:52 -07:00
Konstantin Lebedev
ade94b0d0a
avoid race conditions access to SuperBlock.Version (#3539) 
* avoid race conditions access to  SuperBlock.Version
https://github.com/seaweedfs/seaweedfs/issues/3515

* superBlockAccessLock replace to sync.Mutex
 2022-08-30 00:08:00 -07:00
chrislu
ae6292f9f0 rename variable 2022-08-30 00:07:15 -07:00
Konstantin Lebedev
e16dda88e4
avoid race conditions access to MasterClient.currentMaster (#3538) 
https://github.com/seaweedfs/seaweedfs/issues/3510
 2022-08-30 00:05:38 -07:00
Konstantin Lebedev
105702ebe0
avoid data race access to uploadReaderToChunks.uploadErr (#3550) 
avoid data race access to uploadErr
https://github.com/seaweedfs/seaweedfs/issues/3549
 2022-08-30 00:03:01 -07:00
Konstantin Lebedev
903a06a160
avoid data race access to startDelay (#3548) 2022-08-30 00:02:09 -07:00
chrislu
570b488572 ensure memory alignment on different CPUs 2022-08-29 13:24:12 -07:00
Konstantin Lebedev
4966a3abc7
avoid race conditions access to growRequestCount (#3537) 
https://github.com/seaweedfs/seaweedfs/issues/3511
 2022-08-29 13:23:02 -07:00
Patrick Schmidt
00041f6e08
Fix corrupted mount reads after changes on fh (#3536) 2022-08-29 08:27:13 -07:00
chrislu
b7a887fea1 3.24 2022-08-28 21:53:13 -07:00
Rain Li
089446f5b1
Fix reload page before all file uploaded bug (#3534) 2022-08-28 07:51:47 -07:00
chrislu
cc0c8c5f81 simplify 2022-08-27 00:21:57 -07:00
chrislu
3741779144 minor 2022-08-27 00:20:58 -07:00
chrislu
580a7fa685 adjust for data race 
fix https://github.com/seaweedfs/seaweedfs/issues/3512
 2022-08-27 00:19:13 -07:00
chrislu
87b70a6809 clean up 2022-08-27 00:09:04 -07:00
chrislu
27c7611ebf Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-08-26 23:47:39 -07:00
chrislu
9920d65bc0 gateway to remote object store: adjust upload concurrency 2022-08-26 23:47:37 -07:00
chrislu
c839ce1b19 s3 sink use s3 upload manager 
fix https://github.com/seaweedfs/seaweedfs/issues/3531
 2022-08-26 23:47:12 -07:00
famosss
dc4037925d
fix: Build DeletedVids before reset dn's children (#3530) 2022-08-26 22:52:08 -07:00
chrislu
3f3a1341d8 make CodeQL happy 2022-08-26 17:09:11 -07:00
chrislu
301b49b63f atomic operation 2022-08-26 17:04:11 -07:00
chrislu
57a46f46a0 nano level precision 2022-08-26 16:55:15 -07:00
chrislu
8dae81c5ed Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-08-26 16:49:51 -07:00
chrislu
e0f4366f4c more detailed error 
related to https://github.com/seaweedfs/seaweedfs/issues/3528
 2022-08-26 16:49:48 -07:00
Patrick Schmidt
5df105b1f9
Fix a few data races when reading files in mount (#3527) 2022-08-26 16:41:37 -07:00
chrislu
f5156cf3a8 rename varaible 2022-08-26 10:23:42 -07:00
Konstantin Lebedev
e90ab4ac60
avoid race conditions for OnPeerUpdate (#3525) 
https://github.com/seaweedfs/seaweedfs/issues/3524
 2022-08-26 10:18:49 -07:00
Konstantin Lebedev
4f7a1f67cd
avoid race conditions for diskLocation.MaxVolumeCount (#3526) 2022-08-26 08:41:42 -07:00
chrislu
6f483a4d36 atomic add 
fix https://github.com/seaweedfs/seaweedfs/issues/3514
 2022-08-25 22:20:34 -07:00
chrislu
7394f7feee minor: adjust ordering 2022-08-25 00:20:23 -07:00
chrislu
dbf0de4ce1 minor clean up 2022-08-25 00:19:08 -07:00
Patrick Schmidt
2930263dfd
Fix race conditions during in-flight size checks (#3505) 2022-08-24 20:03:34 -07:00
Patrick Schmidt
7b424a54dc
Add raft server access mutex to avoid races (#3503) 2022-08-24 09:49:05 -07:00
Konstantin Lebedev
f7aeb06544
s3: report metadata if the directory is explicitly created (#3498) 
* replace mkdir to mkFile

* ContentLength must be zero

* revert mkDir

* Seaweedfs-Is-Directory-Key return metadata
 2022-08-24 00:15:44 -07:00
Guo Lei
c57c79a0ab
optimiz commitig compact (#3388) 
* optimiz vacuuming volume

* fix bugx

* rename parameters

* fix conflict

* change copyDataBasedOnIndexFile to an instance method

* close needlemap

* optimiz commiting Vacuum volume for  leveldb index

* fix bugs

* fix leveldb loading bugs

* refactor

* fix leveldb loading bug

* add leveldb recovery

* add test case for levelDB

* modify test case to cover all the new branches

* use one tmpNm instead of two instances

* refactor

* refactor

* move setWatermark to the end

* add test for watermark and updating leveldb

* fix error logic

* refactor, add test

* check nil before close needlemapeer
add test case
fix metric bug

* add tests, fix bugs

* adjust log level
remove wrong test case
refactor

* avoid duplicate  updating metric for leveldb index
 2022-08-23 23:53:35 -07:00
chrislu
10414fd81c ping timeout at 15 seconds 
this 72 minute timeout setting seems unreasonably long

15 seconds is around the time when a new raft leader should be elected.
 2022-08-23 23:28:16 -07:00
askeipx
2e78a522ab
remove old raft servers if they don't answer to pings for too long (#3398) 
* remove old raft servers if they don't answer to pings for too long

add ping durations as options

rename ping fields

fix some todos

get masters through masterclient

raft remove server from leader

use raft servers to ping them

CheckMastersAlive for hashicorp raft only

* prepare blocking ping

* pass waitForReady as param

* pass waitForReady through all functions

* waitForReady works

* refactor

* remove unneeded params

* rollback unneeded changes

* fix
 2022-08-23 23:18:21 -07:00
Guo Lei
762dc219a6
fix deletecount bug (#3499) 2022-08-23 08:53:24 -07:00
chrislu
21a933c932 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-08-23 01:52:32 -07:00