The padding in a connection shard structure

[DifferentialOrange]

There is a 128-byte padding in a connection shard structure and there is no any obvious reason for it.

go-tarantool/connection.go

Line 155 in e1bb59c

_pad [16]uint64 //nolint: unused,structcheck

type ConnShard1 struct {
	rmut     sync.Mutex
	requests [requestsMap]futureList
	bufmut   sync.Mutex
	buf      smallWBuf
	enc      *msgpack.Encoder
	_pad     [16]uint64 //nolint: unused,structcheck
}

type ConnShard2 struct {
	rmut            sync.Mutex
	requests        [requestsMap]futureList
	requestsWithCtx [requestsMap]futureList
	bufmut          sync.Mutex
	buf             smallWBuf
	enc             *msgpack.Encoder
	_pad            [16]uint64 //nolint: unused,structcheck
}

func TestConnShard(t *testing.T) {
	v := ConnShard1{}
	typ := reflect.TypeOf(v)
	fmt.Printf("Type ConnShard1 is %d bytes long\n", typ.Size())
	n := typ.NumField()
	for i := 0; i < n; i++ {
		field := typ.Field(i)
		fmt.Printf("%s at offset %v, size=%d, align=%d\n",
			field.Name, field.Offset, field.Type.Size(),
			field.Type.Align())
	}
	fmt.Printf("ConnShard1 align is %d\n\n\n", typ.Align())

	v2 := ConnShard2{}
	typ2 := reflect.TypeOf(v2)
	fmt.Printf("Type ConnShard2 is %d bytes long\n", typ2.Size())
	n = typ.NumField()
	for i := 0; i < n; i++ {
		field := typ2.Field(i)
		fmt.Printf("%s at offset %v, size=%d, align=%d\n",
			field.Name, field.Offset, field.Type.Size(),
			field.Type.Align())
	}
	fmt.Printf("ConnShard2 align is %d\n", typ2.Align())
}

The output:

Type ConnShard1 is 2240 bytes long
rmut at offset 0, size=8, align=4
requests at offset 8, size=2048, align=8
bufmut at offset 2056, size=8, align=4
buf at offset 2064, size=40, align=8
enc at offset 2104, size=8, align=8
_pad at offset 2112, size=128, align=8
ConnShard1 align is 8


Type ConnShard2 is 4288 bytes long
rmut at offset 0, size=8, align=4
requests at offset 8, size=2048, align=8
requestsWithCtx at offset 2056, size=2048, align=8
bufmut at offset 4104, size=8, align=4
buf at offset 4112, size=40, align=8
enc at offset 4152, size=8, align=8
_pad at offset 4160, size=128, align=8
ConnShard2 align is 8

Originally posted by @vr009 in #173 (comment)

[oleg-jukovec]

f62aac9

We need to calculate the difference for original structures:

const requestsMap = 32
type ConnShard1 struct {
       rmut            sync.Mutex
       requests        [requestsMap]*Future
       first           *Future
       last            **Future
       bufmut          sync.Mutex
       buf             smallWBuf
       enc             *msgpack.Encoder
       bcache          smallBuf
}

type ConnShard2 struct {
       rmut            sync.Mutex
       requests        [requestsMap]*Future
       first           *Future
       last            **Future
       bufmut          sync.Mutex
       buf             smallWBuf
       enc             *msgpack.Encoder
       bcache          smallBuf
       pad            [16]uint64 //nolint: unused,structcheck
}

Type ConnShard1 is 368 bytes long
rmut at offset 0, size=8, align=4
requests at offset 8, size=256, align=8
first at offset 264, size=8, align=8
last at offset 272, size=8, align=8
bufmut at offset 280, size=8, align=4
buf at offset 288, size=40, align=8
enc at offset 328, size=8, align=8
bcache at offset 336, size=32, align=8
ConnShard1 align is 8


Type ConnShard2 is 496 bytes long
rmut at offset 0, size=8, align=4
requests at offset 8, size=256, align=8
first at offset 264, size=8, align=8
last at offset 272, size=8, align=8
bufmut at offset 280, size=8, align=4
buf at offset 288, size=40, align=8
enc at offset 328, size=8, align=8
bcache at offset 336, size=32, align=8
ConnShard2 align is 8

It seems like the last result should be 512. 6 years have passed, perhaps something has changed in the GoLang or I missed something.

Anyway, it doesn't look usefull for the current implementation (and it was broken a long time ago). Let's delete the padding.

[vr009]

I haven't read yet, but FYI https://dave.cheney.net/2015/10/09/padding-is-hard. @DifferentialOrange @oleg-jukovec

I have also heard, that there is a practice in C, to add some byte field after some pointer in the structure and this field is needed for allocation a memory for this pointer, something like that.

[oleg-jukovec]

I haven't read yet, but FYI https://dave.cheney.net/2015/10/09/padding-is-hard. @DifferentialOrange @oleg-jukovec

It doesn't look like our case at first glance.

I have also heard, that there is a practice in C, to add some byte field after some pointer in the structure and this field is needed for allocation a memory for this pointer, something like that.

You could dig deeper and to google something like: "C struct padding CPU cacheline", "C struct padding cache miss", "why array is better than list in C", "cache/memory locality". I can't remember a good article right now, but something like this may help:

https://medium.com/software-design/why-software-developers-should-care-about-cpu-caches-8da04355bb8a
http://psy-lob-saw.blogspot.com/2014/06/notes-on-false-sharing.html
https://habr.com/ru/post/312078/

I assume that the optimization in this case was to avoid intersection of a cache line by nearby connShards inside the array [shards]connShard . As an example, if a thread modifies a field of connShard[X] it should not affect to an another thread that works with a field on connShard[X+1].

This is just a guess, I don't see the point now to find out exactly the reason. It was broken for a long time