Spinning fabric - selvage and scraps

I tried spinning some selvage edges from my quilting and sewing (and some other scrap strips) into fabric yarn.  Turns out it is pretty hard to join fabric strips by spinning, so I had to tie them together.  I'm not sure that the spinning was necessary.  I think you could just tie them together and knit or crochet with it.  I am thinking I will eventually make a rag rug out of this, but I will probably need more yardage for that!