What could these noises be?

I live in a 120 year old house with a chimney, and in all the rooms that border onto the chimney (my bedroom and the bathroom) there is weird noises, like someone pulling a few bricks out of the wall. I have a neighbour which shares the chimney, and she is about 80-85. Its only sometimes, like three or four times a day and theres half an hour full of noises like someone is pulling bricks out of the wall adjacent to mine. It also sounds like they're throwing the bricks down the chimney as well. The chimney is clear and on my side, boarded up. Its not birds, but it could be rats on my neighbour's side. What do you think?